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ABSTRACT 


Efficiency in the acquisition of major defense systems is 
of paramount importance, given today’s environment of 
declining resources. To achieve this efficiency, the project 
office, as the responsible agency for such acquisitions, must 
perform its designated functions both efficiently and 
effectively. The two most important indicators reflecting the 
efficiency and effectiveness of the project office are program 
health and customer satisfaction. The objective of this 
research was to develop an automated self-assessment 
instrument for use by the military project manager to measure 
the program health and customer satisfaction of his/her 
organization. This thesis provides the project manager with 
a viable model depicting the critical organizational design 
factors impacting the program health and internal customer 
satisfaction of the military project office. This study also 
provides the project manager a pilot implementing instrument 
with which to assess the program health and internal customer 
satisfaction of his/her organization. Finally, this research 
effort has produced an independent software application, 
specifically designed to automate the self-assessment process 


within a military project management office. 
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I. INTRODUCTION 


A. PURPOSE 

The purpose of this thesis is to develop a unit self- 
assessment model and a software-based survey instrument to 
effectively measure the program health and customer 


satisfaction of a military project management office. 


B. BACKGROUND 

In July 1994, the Program Executive Office (PEO), 
Tactical Missiles, U.S. Army Material Command (AMC) directed 
all project managers within the PEO to conduct a unit self- 
assessment using the Self-assessment Guide for Organizational 
Performance and Customer Satisfaction, based on the 
Presidential Quality Award criteria. After conducting these 
assessments, two of the Project Managers (PMs), PM Close 
Combat Anti-armor Weapon Systems (CCAWS) and PM Air~to-Ground 
Missile Systems (AGMS), conducted internal reviews of the 
Presidential Award process and criteria and their 
effectiveness in measuring the critical factors which 
determine the overall health of a project office and the level 
of its customers’ satisfaction. These independent reviews 
suggested that the Presidential Award criteria and its 
implementing instrument, the Self-assessment Guide for 
Organizational Performance and Customer Satisfaction, were 
inappropriate to accurately assess and track the level of 


program health and customer satisfaction within a project 


management office. 
Based on these findings, the CCAWS and AGMS Project 
Managers requested that the Naval Postgraduate School develop 


an alternative assessment instrument that will effectively 


assess the factors which directly contribute to the program 


health and customer satisfaction of a project management 


office. 


Cc. THESIS OBJECTIVES 
The objectives of this thesis are to develop: 

ds A unit assessment model and a pilot survey 
instrument that are appropriate and effective for measuring 
the program health and customer satisfaction of a project 
management office. 

2. An automated, network-based user interface, data 
storage, analysis, and presentation application that 


simplifies the unit self-assessment process. 


D. ASSUMPTIONS 


This researcher made the following assumptions concerning 
this research effort: 

1. The Self-assessment Guide for Organizational 
Performance and Customer Satisfaction is an inappropriate 
instrument to measure the program health and customer 
satisfaction within a project management office, for the 
reasons identified by the sponsoring organizations. 

2. The assessment instrument developed as an 
alternative to the Self-assessment Guide for Organizational 
Performance and Customer Satisfaction is only intended for use 
in a military project management office. 

3. The factors contributing to program health and 
customer satisfaction are consistent across project management 
offices. 

4. The sponsoring organizations do not intend to 
significantly modify their computer operating systems within 


the next five year period. 


N 


E. RESEARCH QUESTIONS 

1. Primary 

Does the proposed unit self-assessment package satisfy 
the sponsors’ requirement to efficiently and effectively 
assess and track program health and customer satisfaction? 

2. Subsidiary 

a. Does the proposed assessment address and correct 

the deficiencies of the Self-assessment Guide FOL 
Organizational Performance and Customer Satisfaction, 
identified by the sponsors’ management teams and users? 

(1) Are the assessment items applicable to a 
project management office? 

(2) Is the wording or content of the assess- 
ment items confusing to the intended users? 

(3) Is the assessment too long and time 
consuming? 

(4) Are the assessment items appropriate to 
solicit responses from all personnel within the project 
OFT Ee? 

(5) Does the assessment solicit both quant- 
itative and qualitative feedback? 

b. Is the proposed software application an 
effective method to administer the recommended assessment? 

(1) Is the software application compatible 
with the existing computer resources used by the CCAWS and 
AGMS project management offices? 

(2) Is the software user friendly? 

(3) Is the software application accessible to 
its intended users? 

(4) Does the software enhance the overall 


conduct of the proposed assessment? 


(5) Does the software simplify the user's 
ability to provide quantitative and qualitative feedback? 

(6) Does the software simplify the facili- 
tator's ability to collect, compile, analyze, and present 
generated feedback? 

c. Does the proposed unit self-assessment package 
provide project managers with the information and tools 
necessary to monitor and improve program health and customer 
satisfaction? 

(1) Is the package capable of compiling and 
analyzing data generated from responses in a manner that 
provides useful feedback? 

(2) Are the types and depth of analyses 
included in the package sufficient to meet the managers’ 
requirements? 

(3) Is the package capable of presenting the 
results of generated feedback and analyses in a clear and 
useful manner? 

(4) Is the package capable of storing inform- 
ation, for the purpose of comparison, for a period of not less 
than five (5) years? 

(5) Does the package provide project managers 


with the tools necessary to monitor the improvement of 


identified weaknesses? 


F. ORGANIZATION OF THESIS 


Chapter II of this study details the methodology employed 
by this researcher. It begins by documenting efforts to 
identify an appropriate candidate assessment instrument. It 
then describes the software design techniques used in the 


prototype software development. The chapter concludes with a 


description of the quantitative methods used to evaluate the 
candidate unit self-assessment and the prototype software 
application. 

Chapter III provides background information on the 
military project management office. It begins with a 
description of the military project management environment. 
The chapter continues with a discussion of the major 
functional areas common to most project management offices. 
This chapter concludes by introducing the CCAWS and AGMS 
project offices, the sponsors’ of this thesis. 

Chapter IV addresses the assessment instrument. This 
chapter begins with a discussion of the theory and general 
framework underlying the development and implementation of 
assessment instruments. The chapter then presents a 
theoretical model of organizational dynamics developed by the 
Harvard School of Business. This model is used as the basis 
to compare the organizational models that support the 
assessment instruments addressed in this study. Next, the 
chapter describes and evaluates the two models and assessment 
instruments considered in this study. Discussed first is the 
Presidential Award Model and the Self-assessment Guide for 
Organizational Performance and Customer Satisfaction, used by 
the CCAWS and AGMS Project management Offices in July 1994. 
This is followed by a discussion of the Strategic Leadership 
Planning Model and its assessment instrument, the United 
States Army, Europe (USAREUR) Organizational Diagnostic 
Survey. This model and assessment instrument were selected by 
this researcher as suitable alternatives to the Presidential 
Award Model and Self-assessment Guide for Organizational 
Performance and Customer Satisfaction. Also addressed are 
modifications to the original USAREUR Organizational 


Diagnostic Survey prior to its evaluation. 


Chapter V focuses on the automation of the self- 
assessment process. It begins with a description and 
evaluation of the Continuous Improvement (CI) Toolkit for 
Windows, a commercial software package initially selected by 
this researcher as a solution to automate the self-assessment 
process. Next, the chapter describes the development of an 
independent prototype software program to serve as the user 
interface for the collection of assessment data. The chapter 
concludes with a description of the database storage, 
analysis, and presentation applications designed using 
Microsoft Access. 

Chapter VI begins by presenting the data and qualitative 
responses accumulated about the candidate unit self- 
assessment, assessment items, and the proposed software 
application acquired during the prototype testing. The 
chapter then presents an analysis and discussion of the issues 
related to the Strategic Leadership Planning Model, the 
candidate self-assessment instrument, and the assessment items 
identified during the assessment evaluation. This chapter 
concludes with an analysis of the prototype assessment 
software application and a discussion of anomalies discovered 
in the software during the prototype testing. 

Chapter VII presents conclusions from the data analysis 
and discussion and makes recommendations for implementation to 
the CCAWS and AGMS Project Managers. This chapter concludes 


with recommendations for further study. 


II. METHODOLOGY 


A. INTRODUCTION 

This chapter explains the methods employed in the 
development and analysis of the proposed unit self-assessment 
package. It begins with a description of the qualitative 
methods used to define the research requirements and to obtain 
background information pertaining to the organizational 
assessment process, the military project management office, 
and available assessment instruments and quality improvement 
software. The chapter then focuses on the techniques employed 
in the development of the prototype user interface, data 
storage, analysis, and presentation software applications. 
Next, the chapter describes the quantitative methods used to 
evaluate the candidate self-assessment and the prototype 
software applications. This chapter concludes with a 


description of the limitations applicable to this research. 


B. QUALITATIVE METHODS 

1. Interviews 

Research began with the identification of specific 
research requirements and the justification for the research 
effort. This was accomplished by conducting interviews with 
the CCAWS Project Manager, the AGMS Deputy Project Manager, 
and personnel from within both project management offices, 
identified as participants of the July 1994 unit assessments 


using the Self-assessment Guide for Organizational Performance 


and Customer Satisfaction. 

2. Archival Research 

Once the requirements were defined, a literature review 
was conducted of available materials to locate existing 


assessment instruments and quality improvement software 


programs within the public and private sectors with the 
potential for satisfying the sponsors’ requirements. A second 
literature review was conducted to ascertain background 
information pertaining to the research effort. This review 
focused on the rationale for conducting unit self-assessments 
and the organizational dynamics of a military project 
management office. 

3. Participant Observation 

In addition to conducting literature reviews, this 
researcher observed first-hand the CCAWS and AGMS project 
offices on several occasions totaling a period of one month. 
These visits verified the amount and types of computer 
resources currently in use by the project offices, to include 
identifying potential hardware and software compatibility 
issues. Also observed were the functions of a military 
project management office, its environment, and the 
interaction between its divisions, branches, and product 
lines. 

4. Researcher Educational Preparation 

In February 1995, this researcher attended a four-day 
workshop on the Continuous Improvement (CI) Toolkit for 
Windows, a potential solution to automate the unit self- 
assessment process. The workshop was designed to educate 
system administrators and assessment facilitators in the 
installation, setup, modification, capabilities, and 
procedures of the CI Toolkit. 

In addition to the CI Toolkit, the above workshop focused 
on the organizational assessment process. Specifically, the 
workshop addressed the Malcolm Baldrige Award Criteria, 
preparation and submission of a Baldrige application, and the 


integration of the CI Toolkit into the Baldrige submission 
process. 


This researcher also attended, as part of the Master’s 
degree requirements, a course in strategic management. The 
primary focus of this course was the identification and 
analysis of organizational design factors. Specifically, the 
course presented a theoretical model, developed by the Harvard 
School of Business and modified by Dr. Nancy Roberts, 
depicting the interaction between various inputs, organization 
design factors, and their resultant outcomes. Knowledge 
acquired in this course was applied in the evaluation and 
comparison of the organizational models that support the 
assessment instruments addressed in this study. 

5. Model Construction 

Using the model developed by the Harvard School of 
Business as a basis for comparison, this researcher evaluated 
the organizational models supporting the Self-assessment Guide 
for Organizational Performance and Customer Satisfaction and 
the USAREUR Organizational Diagnostic Survey, the candidate 
assessment instrument selected in this study as a viable 
alternative to the aforementioned Self-assessment Guide for 
Organizational Performance and Customer Satisfaction. The 
results of these comparisons were then used as the basis to 
create a prototype model that more accurately reflects the 
critical organizational design factors of a military project 
management office. 

6. Software Development 

Applying the information gained in interviews with CCAWS 
and AGMS management personnel, an integrated software program 
designed to satisfy the sponsors’ specific automation 
requirements was developed. This program consists of two 
distinctive applications: a user interface and a data storage, 


analysis, and presentation application. 


The user interface application was developed using 
Novell’s AppWare, a graphical, object oriented design and 
development tool. The interface was designed to satisfy four 
of the nine sponsor automation requirements. These 


requirements are: 


° Enable the project office to administer an 
organizational assessment over an existing network 
structure. 


° Provide a familiar, user friendly interface to the 
assessment participants for the purpose of data 
collection. 


e Compatibility with existing computer resources and 
software programs. 


e Export collected data to the analysis and 
presentation application. 


The analysis and presentation application was developed 
using Microsoft’s Access database program. The application 


was designed to address the remaining five user automation 


requirements: 
° Import collected data from the user interface 
program. 
° Analyze collected data. 
° Store collected data for a five year period. 
° Present analyzed data in a meaningful format. 
° Enable data, analysis, and presentation output to 


either a monitor or printer. 


C. QUANTITATIVE METHODS 

1. Prototype Evaluation 

A sample unit self-assessment of both the CCAWS and AGMS 
project management offices was conducted in June 1995 for the 
purpose of evaluating the candidate assessment and the 
prototype software application. The evaluation was 
specifically designed to evaluate the appropriateness of the 


individual functional areas and items of the USAREUR 


10 


Organizational Diagnosti: Survey (modified) for use in 
measuring the program health and customer satisfaction cf a 
project management office and to verify tne structure, 
functionality, and effectiveness of the prototype software 
application (user interface) to collect user feedback. Data 
and feedback from the prototype evaluation were accumulated 
through the use of multiple surveys, discussed below, and 
selected project manager and unit facilitator interviews. 

2. Survey Administration 

Responses were elicited as to the appropriateness of the 
candidate unit self-assessment and the prototype software 


application through the use of four independent surveys. Each 





of the four surveys were completed by the participants of the 
candidate assessment evaluation and prototype software 
testing. The sample used in these surveys and the design of 
each survey are discussed below. 

a. The Sample 

Each of the four surveys were administered to 
personnel participating in the prototype assessment and 
software evaluation, representing the sponsoring organizations 
(CCAWS and AGMS). Every effort was made to obtain a sample 
that accurately represents the demographic makeup of each 
participating organization. Within these guidelines, the 
selection of participating individuals was determined by 
random draw. 

(1) Population. The population from which the 
sample was drawn consisted of all personnel assigned or 
attached to either the CCAWS of AGMS project offices. The 
AGMS project office employs 97 personnel, divided into four 
division and 17? separate branch offices. CCAWS employs a 
total of 103 personnel in four divisions, separated into 15 


branch offices. The total population numbered 200 personnei. 


fee 


This population was divided by the researcher into three major 
demographic categories: level of responsibility, sex, and 
employment status. The level of responsibility category was 
sub-divided into management personnel, clerical personnel, and 
all others. Sex was further divided into male and female 
participants, and employment status was broken down into 


military personnel and civilian employees. The demographic 


break-down of the population is presented in Table 1. 


MGT Clerical Other Male Female 
12 l 74 6 32 
1 





] 2) 
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Table 1: The Population. 


(2) The Sample. The sample target was 50 
participants, 25% of the total population, consisting of an 
equal percentage from each demographic category defined above 
and a proportionate number from each of the participating 
organizations. The demographic make-up of the actual sample, 
by total and percentage, is presented in Table 2. 

The accumulated sample is representative of the 
CCAWS/AGMS population, with the following exception: The 
sample contains a disproportionate number of military 
managers. This is due to unavoidable scheduling conflicts and 


the researcher’s decision to select alternates from within 


these categories, when available, to maximize input from these 


categories with respect to Survey #2. 


Male Female 
17 8 














100% 





Table 2: The Sample. 


b. Survey #1 
Survey #1 (Appendix A) is the USAREUR Organizational 
Diagnostic Survey (modified). Researcher modifications to the 
original USAREUR Organizational Diagnostic Survey (Appendix B) 
preceding the assessment evaluation and prototype testing are 
discussed below. This survey, in its final form, upon 
completion of the researcher’s analysis is the CCAWS/AGMS Unit 
Self-assessment Survey (Appendix C), a deliverable product of 
this thesis. 
(1) Survey Design. The USAREUR Organizational 
Diagnostic Survey (modified) is composed of seventy-four (74) 
items, grouped into eight functional areas and a general 


category. The functional areas represented in the survey are: 


e Strategy 
e Tasks 

e Structure 
e People 
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° Rewards 


° Processes 
° Leadership 
° Technology 


This survey was designed to measure each respondent’s opinion 
as to the program health of his/her respective organization, 
with respect to each functional listed above. 

(2) Survey Modifications. Several mod- 
ifications to the original USAREUR Organizational Diagnostic 
Survey were made prior to conducting the assessment evaluation 
and prototype testing. These modifications were made to 
accomplish the following items: 


° Separate items soliciting multiple responses into 


individual items. 
° Clarify items with complex or ambiguous wording. 


Remove items clearly inappropriate for use ina 
project management office. 


Replace all references to the “organization” with 
the “project management office”. 


(3) Analysis. The analysis of data measuring 
the health of the participating organizations is beyond the 
scope of this thesis and therefore will not be presented as 
part of this study. These data elements were collected solely 
for the purposes of establishing an initial baseline to be 
used by each participating unit for future assessments and to 
accumulate realistic data for use in the design of data 
queries, analysis tools, and presentation layouts. 

c. Survey #2 
Survey #2 is the Unit Self-assessment Item Survey. 
(Appendix D) 

(1) Survey Design. Survey #2 was designed to 

solicit participant feedback about each individual item 


incorporated in the USAREUR Organizational Diagnostic Survey, 
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as modified by this researcher. Respondents were asked to 
rate six different aspects measuring three general categories 
of each item. The three categories were: 


° The wording of the item 


a 
t) 


[The item’s content 





° The scale provided to answer the item 

With respect to the item’s wording, respondents were asked to 
rate the wording’s clarity and conciseness. As for the item’s 
content, respondents were asked to rate the understandability 
of the item, the relevancy of the content to their job 
position, and the suitability of the item. In this context 
suitability is defined as _ unoffensive. Lastly, each 
respondent was asked to rate the appropriateness of the scale 
provided to respond to the item. 

(2) Analysis. Respondents were asked to rate 
each aspect described above using a bipolar scale with three 
intermediate points, five points total (Figure 1). The 
endpoints of each scale were labeled with opposing adjective 
descriptors connoting a positive or negative response. Each 
respondent was also asked to provide comments in these areas 


on each item. 


Is the item’s wording: 


Oo Oo Oo O = 


| nd 





Ambiguous Clear 
Figure 1: Example Bipolar Survey Item. 


After all data were collected, individual 
responses were then translated to numerical values on a scale 


from negative two to two, zero being a neutral response. The 


numerical values were then grouped by item and aspect. 





Calculations were performed to determine percentage response 
of the maximum possible score (100) per item, and the mean 
response and standard deviation for each item, by aspect. 
Items were then reviewed using the percentage of the maximum 
possible score calculations to identify questionable items. 
The screening criteria for this review was 99 percent. Items 
identified as questionable were then analyzed with respect to 
each measured aspect to identify those aspects containing 
significant shortcomings. A significant shortcoming was 
defined as an item aspect receiving less than a mean score of 
1.91, which corresponds to an effectiveness rating of less 
than 95%. Items with one or more significant shortcoming were 
then analyzed against the qualitative responses for each 
deficient aspect to determine the appropriate corrective 
action. 

d. Survey #3 

Survey #3 is the Unit Self-assessment Survey, listed 
as Part A of the Unit Self-assessment and Software Survey. 
(Appendix E) 

(1) Survey Design. Survey #3 was designed to 
elicit data and qualitative feedback on the candidate self- 
assessment’s usefulness, applicability, and procedures, as 
well as general comments and suggestions for improvement. 
This survey employed two different questioning techniques: the 
bipolar method, described in survey #2, and open items 
eliciting a qualitative text or numerical response. 

(2) Analysis. Bipolar items, as in survey #2, 
asked respondents to rate each aspect described above of each 
item using a scale with three intermediate points. Again, the 
endpoints of each scale were labeled with opposing adjective 
descriptors connoting a positive or negative response. 


Responses to these items were analyzed in a similar manner as 
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those described in survey #2. The difference being that there 
was no requirement to group individual items to conduct an 
initial screening for potential weaknesses. 

Open items were designed to enable each 
respondent to provide his/her opinion, comments, or 
suggestions concerning the candidate assessment and its 
procedures. Responses to these items were summarized for use 
in the analysis and discussion of the overall assessment 
process. 

e. Survey #4 

Survey #4 is the Unit Self-assessment Software 
Survey, listed as Part B of the Unit Self-assessment and 
Software Survey. (Appendix E) 

(1) Survey Design. Survey #4 was designed to 
solicit feedback on the self-assessment software’s usefulness, 
applicability, and procedures, as well as general comments and 
suggestions for improvement of the software. As with Survey 
#3, this survey employed two different questioning techniques: 
the bipolar method, and open items eliciting a qualitative 
response. 

(2) Analysis. The analysis of the items in 


this survey is identical to that described for survey #3. 


D. LIMITATIONS 
The following limitations apply to this research effort: 
1. This thesis is limited to the study of an assessment 
instrument for use in a military project management office. 
2. The data and qualitative responses used in the 
analysis of the candidate assessment instrument and the 
prototype software application are limited to inputs from the 


CCAWS and AGMS project management offices. 
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3. Software applications developed as the result of this 
study are the sole property of the United States Government 
and can only be modified with express permission of the CCAWS 
or AGMS Project Managers. 

A. Modifications to software developed using Novell’s 
AppWare requires the purchase of the AppWare development 
software and obtaining application project files that are 
external to the program. These project files have been 
retained by the researcher and the CCAWS and AGMS project 


management offices. 


E. SUMMARY 

The purpose of this research effort was to develop an 
automated unit self-assessment package appropriate to measure 
the program health and customer satisfaction of a military 
project management office. Using both qualitative and 
quantitative research techniques, this researcher has 
evaluated the unit assessment previously used by the CCAWS and 
AGMS project management offices against user specific 
requirements; identified the shortcomings with the previous 
assessment; selected an existing alternative as a candidate 
assessment; tested, analyzed, and modified the candidate 
assessment; automated the assessment process; and provided 
recommendations the CCAWS and AGMS Project Managers for the 


implementation of the proposed assessment and software 


package. 
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III. THE PROJECT MANAGEMENT OFFICE 


A. INTRODUCTION 

A Project Management Office (PMO) is an organization 
responsible to facilitate the design, development, production, 
fielding, and support of a major systems acquisition. The 
purpose of this chapter is to provide background information 
about the military project office as a customer of the unit 
self-assessment. This discussion is divided into four areas: 
The military project management environment; the project 
office structure; functions of the project office; and a 
description of the CCAWS and AGMS project offices. 


B. ENVIRONMENT 


The environment in which the PMO exists is quite complex. 


Cm 
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Figure 2: The Project Management Environment. 
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Within this environment, depicted in Figure 2, are many 
agencies, both within and external to the Department of 
Defense, with which personnel within the PMO must directly 
interact with on a daily, weekly, or monthly basis to ensure 
the successful accomplishment of the project’s mission. In 
addition to their coordinating role, many of these agencies 
provide guidance or exert a significant amount of control over 
the project office’s objectives and activities. The project 
office must proactively respond to the needs and desires of 


these agencies for a number of varying reasons. These reasons 


include: 
° Congressional support. 
. Public support. 
e Resources. 
e Legal Support. 
. Task accomplishment. 


C. STRUCTURE 

The military project management office is structured 
along the traditional functional and commodity area lines. 
Many project offices operate, however, along product lines in 
Integrated Product Teams (IPTs) to enhance the efficiency and 
effectiveness of the services they perform. Viewing the 
project office’s structure as it relates to the implementation 
of its IPTs, the project office becomes a matrixed 
organization (Figure 3). Under this concept, selected 
personnel assigned within the organization’s functional or 
commodity divisions are aligned to provide dedicated support 
in their specialty towards the development of one or more of 
the project office’s specified products. IPTs are normally 
lead by the manager with responsibility for the completed 


product. Members of the IPT remain assigned in their 
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functional or commodity area, but are responsible to the IPT 


team leader for product related activities. 
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Figure 3: Example Matrix Organization. 


D. FUNCTIONS 


Each military project office is uniquely structured and 
differs in its approach in supporting the development, 
production, and fielding of its assigned weapon system. 
Despite these differences, all military project offices 
perform the same general or “core” functions. These functions 
and their integration with each other are depicted in Figure 
4. (Cleland, Gallagher, and Whitehead, 1993, p. 10.21) The 


“core” functions identified by these authors include: 
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° Project Management 


° Contract Management 

° Business Management 

° System Engineering Management 

e Integrated Logistic Support Management 
e Production Management 

® Configuration Management 

° Risk Management 
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- Figure 4: Project Management Functional Integration. 


Project Management is the function of integrating each of the 
projects subordinate functions for the successful development 
of the desired system. This integration is facilitated 
through the business, administrative, and technical management 
of the sequence of project activities in the project cycle. 
Contract Management or acquisition management is a project 
function dedicated to the planning, design, implementation, 
and administration aspects of contractual obligations and 
documents involved in the acquisition of a major system. 
Tasks included in this function include: 


° Procurement planning and guidance. 


2? 


. Developing contract requirements. 

° Contract administration. 
Business Management encompasses those activities normally 
associated with the efficient functioning of all 


organizations. Categories within this function include: 


o Financial Management 
° Personnel Management 
e Resource Management 
. Time Management 


System Engineering Management is the practical application of 
scientific and engineering methods and techniques to: 


. Transform mission need and operational requirement 
statements into a description of a system 
configuration which best provides the required 
capabilities in accordance with the specified 
effectiveness metrics. 


. Coordinate the system design definition and detail 
design process. 


° Integrate all technical considerations and assure 
compatibility of all physical, functional, and 
technical interfaces to facilitate an optimized 
total system design. 


. Integrate the activities of all the applicable 
engineering disciplines and specialties into a 
unified engineering team approach to facilitate an 
effective and efficient system design process. 


Integrated Logistic Support (ILS) Management designs and 
develops the logistic support system concurrent with the 
objective system development to ensure that: 


* The supportability considerations influence the 
system design process. 


) ILS impacts are addressed during the system trade 
studies in the early project phases. 


e ILS options and tradeoffs can be considered before 
the system design is frozen or the product baseline 
are established. 


. The ultimate logistic system is an integrated 
system with congruence among all logistic elements. 
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Production Management includes those activities required to 
produce the total system effectively and efficiently. Tasks 
in this are include: 

° Production planning and support. 


e Production and industrial engineering for 
process, facilities, and plant equipment design, 
development, and integration. 


e Production inventory management and control. 
Configuration Management is the project office function 
responsible for documenting the functional and physical 
characteristics of the objective system configuration. Tasks 
encompassed within this function include: 


° Technical definition and documentation of the 
system functional and physical characteristics. 


° Technical and administrative control of all changes 
to the systems characteristics and attributes. 


° Recording and reporting change processing and 
implementation status. 


Risk Management is the function designed to force organized 
purposeful thought to the subject of eliminating, minimizing, 
or containing the effects of undesirable occurrences. Risk 
management is a continual process throughout the acquisition 
cycle. Typical risks associated with the acquisition of a 


major weapon system include: 


° Technical Risk 

° Programmatic Risk 

° Supportability Risk 
° Schedule Risk 

. Cost Risk 


The project manager attempts to eliminate or minimize the 
occurrence of risk by: 
° Identifying and planning for potential risk. 


° Assessing and developing Courses of Action (COA) to 
address risk. 
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° Analyzing and comparing each COA, then selecting 
the best suited to mitigate risk. 





° Implementing the selected COA to avoid, control, 
assume, or transfer risk. 


E. THE CLOSE COMBAT ANTI-ARMOR WEAPON SYSTEMS AND THE AIR- 
TO-GROUND MISSILE SYSTEMS PROJECT OFFICES 


Within the Army acquisition structure, project management 


offices are grouped into programs. Each program or Program 
Executive Office (PEO) reports directly to the Assistant 
Secretary of the Army for Research, Development, and 


Acquisition. Currently there are ten such programs, each 


encompassing the development of major systems that represent 
a unique capability on the modern battlefield 
communication, etc.) 


(i.e. missiles, 
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Figure 5: The Location of the CCAWS and AGMS Project Offices 
within the Army Acquisition System Structure. 
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The CCAWS and AGMS Project Management Offices are assigned to 
the PEO, Tactical Missiles, located at Redstone Arsenal, 
Alabama (Figure 5). 

1. Air-to-Ground Missile Systems (AGMS) 

AGMS is responsible for the planning, direction, and 
control of tasks involved in the development and upgrade of 
the Hellfire, Hellfire II, and Longbow Missile Systems. This 
includes all phases of development, procurement, production, 
distribution, and logistics support for the purpose of 
maintaining balanced programs to accomplish the stated 
objectives of the Army Acquisition Executive. The AGMS 


organizational structure is depicted in Figure 6. 















PRODUCT 


\, sees 


LONGBOW 
ap doncien >in) 
Pa/TEST/ J 
| Mor ) CONFIGURE PROGRAM 
MGT ean 









DEPUTY 
PM 











\, 
——— : 
/ WELLFIRE 
(ereticarion 
SS 
Csseane\> 
(INTEGRATION Wi 





x 
\ acquisition STS ENGR 


yy 











BUs/ 
LAUNCHER SOFTWARE 


Comepanasy y - ( Connigure ae 
b = HARDWARE \ ANALYSIS : 


Faas 


i narionat 
\ Le 














CONTRACTOR 
SUPPORT 


— 


a | 
COMPUTER 
SUPPORT 





Cy ome 
Cc) = 
Cj) «= 


cre 














Figure 6: AGMS Organizational Diagram. 
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The AGMS project office also exercises operational control 
over personnel assigned to U.S. Army Missile Command (MICOM) 
functional organizations collocated with the project office to 
provide support in the areas of logistics, product assurance, 
engineering, test, and configuration management. 

2. Close Combat Anti-armor Weapon Systems (CCAWS) 

CCAWS is responsible for the development and upgrade of 
the Tube launched, Optically tracked, Wire-guided Weapon 
System (TOW), to include the Improved Target Acquisition 
System (ITAS), the Improved Bradley Acquisition System (IBAS), 
and all TOW missile configurations. CCAWS, as with AGMS, 
exercises operational control over personnel assigned to MICOM 
functional organizations collocated with the project office. 


The CCAWS organizational structure is depicted in Figure 7. 
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Figure 7: CCAWS Organizational Diagram. 
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IV. THE INSTRUMENTS 


A. INTRODUCTION 

The purpose of this chapter is to present and evaluate 
the assessment instruments addressed in this study and their 
supporting models. The chapter begins with a brief discussion 
of the theory and conceptual framework underlying the 
development and implementation of organizational assessments. 
It then presents a conceptual model developed by the Harvard 
School of Business, to analyze the effectiveness and 
interaction of organizational components. The chapter 
concludes with the description and evaluation of two 
assessment models and their assessment instruments used to 
measure the program health and customer satisfaction in a 
project management office. The first of these models and 
assessment instruments were used by the CCAWS and AGMS project 
offices to conduct assessments of their respective 
organizations in July 1994. The second model and assessment 
instrument were selected during the literature review as 
viable alternatives to those used by CCAWS and AGMS in July 
1994. 


B. THEORY AND GENERAL FRAMEWORK 

The measurement of an organization’s productivity and 
customer satisfaction is a crucial managerial task, both in 
the public and private sectors, given today’s environment of 
declining resources. In order to effectively manage their 
scarce resources, managers must develop appropriate 
instruments to assess and improve the efficiency of their 
organization. The theory underlying the development and 
employment of such measurement instruments is termed Total 


Quality Management. 


1. Total Quality Management 

a. Philosophy 

Total Quality Management (TQM) is a philosophy, as 
well as a set of guiding principles, that represents the 
foundation of continuous improvement in organizations. Et 
applies both qualitative and quantitative methods to improve 
the materials and services provided by an organization, and 
seeks to meet the customer's needs, both now and in the 
future. The application of TOM integrates fundamental 
management techniques, existing improvement efforts, and 
technical tools in an effort to continually improve all 
organizational processes. 

TOM addresses the quality of management as well as 
the management of quality. It involves everyone in the 
organization in a long term endeavor to develop processes that 
are customer oriented, flexible, responsive, and constantly 
improved in quality. Since quality includes any factor of a 
product or service of value to a customer, TQM seeks to create 
an environment to sustain a culture committed to the 
continuous quality improvement of these factors. TQM is also 
a means for improving personal effectiveness and performance, 
and for aligning and focusing all efforts throughout the 
organization. (Mansi, 1989) 

b. Quality Measurement 

As TQM principles are implemented within an 
organization, the measurements of quality and customer 
satisfaction become crucial elements in achieving improved 
productivity. Quality results from the successful integration 
of all an organization’s functions to deliver first rate 
products and services. Customer satisfaction is the result of 
continuous quality control of the delivered products and 


services. Since all organizational functions play a part in 
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quality performance, each critical function within the 
organization should be measured to determine the level of 
quality at which the organization is functioning. 

Civilian organizations measure quality performance 
using several methods. One method use is Statistical Process 
Control (SPE). Organizations using this method collect 
performance data, such as rework or defect rates, that focus 
on the production function of the organization. Although this 
is a valid method, it is narrow in focus and does not reflect 
organization-wide quality management efforts. Another form of 
measurement that is used, and one that does more readily 
identify the impact of organization-wide quality management is 
measurement based on actions. That is, measurement techniques 
that record customer feedback on products, measure the number 
of customer complaints, or measure other such external data. 
This method is also narrowly focused and fails to address 
quality issues internal to the organization. Still, other 
organizations measure financial performance to gage quality 
within their organization. This form of quality measurement 
does not take into account customer satisfaction, nor does it 
fully consider organizational management techniques and other 
critical factors. (Saraph, Benson, and Schroeder, 1989) 

Another quality measurement instrument used by 
civilian organizations is the organizational self-assessment. 
This type of measurement is considered by many to be the most 
comprehensive technique currently used in organizations. The 
use of self-assessment instruments provides an organization’s 
management with increased flexibility, allowing management to 
modify the assessment instrument, as needed, to reflect the 
organization’s changing environment, values, and procedures. 
The most widely used and popular self-assessment instrument is 
the Malcolm Baldrige National Quality Award. The Baldrige 


award was created to recognize the achievements of U.S. 
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companies that excel in the implementation of successful 
quality strategies. It is a highly respected accolade and a 
useful diagnostic tool that measures the organization’s 
overall quality management and performance. The Baldrige 
Award incorporates a team based, consensus approach by the 
organization’s management to measure of a set of seven quality 
criteria, which are in turn used to improve the organization’s 
performance. (ASQC, 1995) 

Just as the use of TQM concepts have become the 
vehicle for improving productivity and customer satisfaction 
in all sectors of private industry, so have they become the 
impetus for improvement in Federal organizations. Subsequent 
to the onset of "quality consciousness" within the private 
sector, President Clinton encouraged Federal managers at all 
levels to manage quality improvement in accordance with 
criteria established by the Presidential Award for Quality. 
These criteria, extracted and modified from the Baldrige 
Award, guide Federal agencies in striving toward increased 
quality management by improving operational performance and 
customer satisfaction. The National Performance Review Report 
of September 1993 reinforced this viewpoint suggesting that 
all Federal agencies adopt these criteria and conduct regular 
assessments using the Presidential Award’s implementing 
instrument, the Self Assessment Guide for Organizational 
Performance and Customer Satisfaction, as a means of 
identifying areas needing improvement. (Federal Quality 
Institute, 1993) 

Recalling the principal focus of this study, which 
is to develop an assessment instrument to effectively measure 
the program health and customer satisfaction of a project 
management office, one must begin by analyzing the assessment 
instrument currently being used, as well as_ viable 


alternatives to the same. A study of these assessment 
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instruments, however, would be fruitless without first 
considering the basis for the instrumentation design. 
Therefore, this study will first address the assessment models 
supporting each of the instruments to be considered, followed 
by a discussion of the assessment instruments themselves. In 
order to effectively analyze and compare the models identified 
in this study, the Harvard Business School model will be used. 
The Harvard Business School model was chosen as the conceptual 
basis for comparison because it is the most comprehensive tool 
available for use in analyzing the interrelationships between 
various critical organizational activities. 

2. The Harvard Business School Model (modified) 

The Harvard Business School model was developed as an 
analysis tool to educate its students in the factors involved 
in strategic planning and management. The model has 
subsequently been adopted and used by Dr. Nancy C. Roberts for 
instruction at the Naval Postgraduate School, Monterey, 
California. In addition to teaching this model, Dr. Roberts 
has successfully applied the model, as a consultant, while 
analyzing the strategic planning factors of numerous private 
sector, state government, and Federal Government 
organizations. Based on her exposure to this model, through 
research and practical application, Dr. Roberts has made 
several modifications to the original Harvard model. The 
Harvard model, as modified by Dr. Roberts (Figure 8), is used 
by this researcher as the basis to evaluate the models that 
support the assessment instruments addressed in this study. 
Each model is evaluated against the Harvard model to determine 
the comprehensiveness in which it addresses the critical 
organizational design factors that determine the success or 


failure of an organization. 


33 


Strategic Inputs 












Environment 
(Context) 


Strategic Implementation 


Key Success 
Factors 


Direction Setting 


Mission Measurement & Control Systems 


Training & Development 


Vision 


v Reward Systems 
Strategic Issues Recruitment & Selection Systems 


Strategies 
1 i] 


PidaleMORGCtives lire iiss ied areas pes eels eas eine meal mer sls ne ie re See 





Figure 8: The Harvard Business School Model 


The Harvard Business School model (modified) 


analysis in four general areas: 


° Strategic Inputs 

° Direction Setting 

° Strategic Implementation 
° Strategic Evaluation 


Strategic Evaluation 











Culture and 
Management 
Style 


Problems 


UY 


(modified). 


focuses its 


The interrelationships between these areas and their internal 


elements are indicated with solid arrows. 
represent feedback from one area to another. 
The first 


area, Strategic Inputs, 


categories: Environmental Factors, 
Environmental Factors - 
organization of which it has no direct control. 
environmental factors include: 
level of competition in the industry, 


cnange. 


34 


consists 


The status of the economy, 


Dashed arrows 


of two 


and Key Success Factors. 


are defined as elements that affect an 


Examples of 
the 


and social and political 


Key Success Factors - are critical elements required for an 
organization to succeed in a particular endeavor, given the 
aforementioned environmental factors. Examples of key success 
factors are: Responsiveness, Marketing, and the Delivery of 
Goods or Services. 

The second area addressed by the Harvard model is 
Direction Setting. The components within this area 
collectively represent top level management’s plan for the 
organization to accomplish its key success factors, given the 
current environmental factors. The elements of direction 


setting include: 


° The Mission 
° The Vision 
° Strategic Issues 
° Strategies 
° Goals and Objectives 
The Mission - is a general statement describing the 


organization’s overall purpose or reason for being. 

The Vision - of an organization conceptualizes the desired 
endstate of the organization at mission accomplishment. 
Strategic Issues - are items of critical focus, determined by 
senior management, that relate to the successful 
accomplishment of the organization’s stated mission and key 
success factors. 

Strategies - are management’s overarching plans formulated to 
address the identified strategic issues. 

Goals - are defined as interim success points established by 
management. Goals are used to measure the organization’s 
progress towards accomplishing their defined strategies. 
Objectives - are the basic building blocks or steps that 


comprise the organization’s established goals. 


39 


The third area addressed by the Harvard model is 
Strategic Implementation. This area is focused on _ the 
identification and assessment of the organization’s design 
factors, as well as their interaction with one another. 


Organizational design factors, according to the Harvard model 


include: 
° Task Requirements 
° Technology 
e People 
° Organization Structure 
° Process/Systems 
Task Requirements - are defined as the basic activities an 


organization must perform effectively to accomplish its 
mission and achieve its key success factors, given the 
environmental factors. 

Technology - is defined as the basic methods or techniques 
used within the organization. 

People - refers to all human resources available to the 
organization. 

Organization Structure - focuses on the basic grouping of the 
organization’s activities and people. Structure is also 
concerned with the devices used by the organization to 
integrate these activities and people. 

Processes/Systems - is focused on the various processes and 
systems in place within the organization to guide the 
remaining organizational design factors towards the 
achievement of the key success factors and mission 


accomplishment. Critical processes/systems identified by the 
Harvard model include: 


. Measurement and Control 
. Training and Development 
° Rewards 


e Recruitment and Selection 
The final area evaluated by the Harvard model is 


Strategic Evaluation. This area is comprised of: 


° Culture and Management Style 
° Outputs 
® Outcomes 
* Problems 
Culture and Management Style - is the pattern of past 


behavior, activity, and managerial effectiveness which may 
have an affect on the current or future functioning of the 
organization. 

Outputs - are clearly identifiable and measurable products or 
services provided by an organization. 

Outcomes - are the consequences or results affecting the 
organization or its environment related to the organization’s 
outputs. 

Problems - are defined as critical flaws or malfunctions 
associated with one or more of the previously defined 


categories. 


C. INSTRUMENT EVALUATION 
1. The Presidential Award for Quality and the Unit Self- 
Assessment Guide for Organizational Performance and 
Customer Satisfaction 
a. The Presidential Award for Quality 
The Presidential Award for Quality was instituted in 
1988. The award was created to recognize organizations that 
have implemented quality management in an exemplary manner, 
resulting in high quality products and services, the effective 
use of taxpayer dollars, and to promote quality management 
awareness and implementation throughout the Federal 


Government. (Federal Quality Institute, 1995, p. 1) 
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The Presidential Award model addresses seven quality 
criteria, common to most Federal organizations. These 
criteria (Figure 9) were extracted and modified from the 
Baldrige model to guide Federal agencies in striving toward 
increased quality management, improved operational 
performance, and customer satisfaction. As with the Harvard 
model, the relationships between each of the areas are 


indicated with solid arrows. 
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Figure 9: The Presidential Award Model. 


According to the Presidential Award criteria: 
Leadership - is defined as the executives’ personal commitment 
and involvement in creating and sustaining an organizational 
vision and customer focus orientation, as well as clear and 
visible quality values. The evaluation of this criterion is 
concerned with how the vision, values and customer focus 
orientation are integrated into the management system, labor 


relations, and external partnerships. 
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Management of Process Quality - addresses the systematic 
processes used by the organization for continuous improvement 
of quality and operational performance, the design and 
management of process quality for all work units, management 
of internal customer-supplier relationships, supplier and 
intermediary quality, and quality assessment. 

Human Resource Development and Management - examines how the 
entire workforce is enabled to develop its full potential, and 
to pursue quality and operational performance improvement 
goals. It also addresses efforts to build and maintain an 
environment for workforce excellence. 

Strategic Quality Planning - focuses on the organization’s 
planning process, both short and long-term, and how key 
quality requirements are integrated into the overall planning 
process. 

Information and Analysis - examines the scope, management, and 
use of data, information and measures, and how they are used 
to drive quality and operational performance improvement. It 
also examines the adequacy of the organization’s data, 
information, and analysis system to support the improvement of 
customer satisfaction, products, services, and processes. 
Customer Focus and Satisfaction 7 encompasses the 
organization’s knowledge of external customer requirements, 
the techniques used to establish and maintain customer 
relationships, and the methods used to determine customer 
satisfaction. 

Quality and Operational Results - considers the organization’s 
trends and quality levels for products and = services, 
operational performance, business processes and support 
services, supplier and intermediary quality, and 


comparison/benchmark data. 


Ww 
9) 





Evaluating the above criteria against the Harvard 
Business School model, we find that the Presidential Award 
Criteria, although appropriate to assess the quality 
improvement of an organization, does very little in the way of 
addressing the interaction and effectiveness of its 
organizational components. A comparison of the two models 
(Figure 10) reveals that the factors in the Harvard model that 


are also addressed by the Presidential model are limited to: 


. Mission, Vision, Strategic Issues, Strategies, and 
Goals/Objectives 

° Process/Systems 

° People 

° Outputs 

. Outcomes 
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Figure 10: A Comparison of the Presidential Award Model and 
the Harvard Business School Model. 
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In the above diagram, factors that are similar in both models 
are connected with solid arrows. Factors in the Harvard model 
that are not addressed by the Presidential model are 
represented with darkened boxes. 

Even though the factors listed above are addressed by 
both the Harvard model and the Presidential model, the clear 
focus of the Presidential Award criteria is to measure the 
quality improvement efforts in each of these areas, rather 
than their individual effectiveness and interaction with one 
another. Therefore, the Presidential Award model is not the 
most appropriate model for use in measuring both the program 
health and customer satisfaction of an organization. 


b. The Self-assessment Guide for Organizational 
Performance and Customer Satisfaction 


To assist Federal organizations in conducting a unit 
self-assessment prior to submitting an application for the 
Presidential Award, the Unit Self-assessment Guide for 
Organizational Performance and Customer Satisfaction, based on 
the award’s criteria, was developed and made available to all 
Federal organizations. The self-assessment consists of 66 
individual items, grouped in categories aligned with the 
Presidential Award criteria. This instrument, as with the 
Baldrige Award, incorporates a team _ based, consensus 
management approach to evaluate each item. 

The Self-assessment Guide for Organizational 
Performance and Customer Satisfaction was designed to be 
universally applied to all Federal organizations. However, 
after conducting a self-assessment using the Self-assessment 
Guide for Organizational Performance and Customer Satisfaction 
in July 1994, the management teams and assessment participants 
from the CCAWS and AGMS project management offices identified 
several shortcomings with the instrument and its use in 


measuring the critical organizational factors within a project 


AL 





management office. The shortcomings identified were: 
® The assessment was lengthy and time consuming. 
° The assessment items were wordy and ambiguous. 


° Many assessment items did not apply to a project 
management office. 


The assessment did not address several critical 
functional areas within a project management 
office. 


® The assessment did not provide a vehicle to collect 
qualitative feedback. 


Given the above shortcomings and the fact that this assessment 
was constructed using the Presidential Award criteria as a 
basis, this researcher concludes that the Unit Self-assessment 
Guide for Organizational Performance and Customer 
Satisfaction, as with the Presidential Award Criteria, is 
inappropriate for use in measuring the program health and 
customer satisfaction of a project management office. 


2. The Strategic Leadership Planning Model and the 
USAREUR Organizational Diagnostic Survey 


a. The Strategic Leadership Planning Model 

Based on the shortcomings identified with the 
Presidential Award for Quality and the Self-assessment Guide 
for Organizational Performance and Customer Satisfaction, a 
literature review was conducted to locate existing models and 
unit self-assessment instruments, within the public and 
private sectors that meet the project managers’ requirements 
as potential alternatives to the above model and assessment 
instrument. Of the available models and assessment 
instruments, the Strategic Leadership Planning model and the 
United States Army, Europe (USAREUR) Organizational Diagnostic 
Survey, based on the Strategic Planning model, were selected 
as the model and instrument whose framework most accurately 
reflect the critical factors, identified by the CCAWS Project 
Manager and the AGMS Deputy Project Manager, that determine 


42 


the program health and customer satisfaction in a project 
offied. 

The program health of a project management office is 
defined by these managers as: 


The efficient functioning of an organization 
that results in the successful fielding and support 
of material to its designated users, where the 
organizational climate is based on teamwork and is 
one in which subordinates are empowered and 
resourced to accomplish their assigned tasks; 
decisions are made at the lowest appropriate level 
within the organization; personnel enjoy their 
working environment; and workers have confidence in 
the products and services they deliver. (Interviews, 
1995) 


Customer satisfaction, for the purpose of this 
study, is defined as: 


An organization’s demonstrated success in 
meeting the requirements of the recipients to which 
services are performed or products are delivered by 
the organization. (Hunt, 1993) 


The Strategic Leadership Planning model (Figure 11) 
addresses four general factor categories. (DA, CGSC, Strategic 


Planning, 1994) These categories are: 


e Inputs 

° Organizational Strategy 

° Organizational Design Factors 
° Outputs 


Included in the Inputs category are factors external to 
and within an organization that are present and necessary to 
be considered at the outset of the strategic planning process. 
These Factors, according to the Strategic Leadership Planning 


model, are: 


. Purpose 

° Environment 
e Culture 

s Expectations 
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Figure 11: The Strategic Leadership Planning Model. 


° Demands 
° Opportunities 
Purpose - is defined as a broad, general definition of the 


organization’s reason for being in existence. 

Environment - as in the Harvard model, encompasses those 
elements affecting the organization that are beyond the direct 
control of the organization’s management. 

Culture - is the pattern of past behavior, activity, and 
effectiveness which may have an affect on the current or 
future functioning of the organization. 

Expectations - are standards or achievements, internal and 


external to the organization, considered reasonable, due, or 
necessary to be obtained. 


Demands - are defined as requirements of work or of the 


expenditure of resources. 
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Opportunities - are favorable junctures in circumstances that 
produce a good chance for advancement or progress. 

The organizational strategies category encompasses 
factors that comprise top management’s overarching plan for 


operating the organization. These factors include: 


° Values 

° Mission 

° Goals 

° Objectives 


Values - are defined as those core beliefs held by management 
that are assimilated into and guide the formulation of the 
organization’s mission, goals, and objectives. 

The mission, goals, and objectives factor definitions are 
similar to the factors of the same names previously defined in 
the description of the Harvard Business School model. 

The next category addressed by the Strategic Leadership 
Planning model is Organizational Design Factors. Factors in 


this category include: 


° Technology 
e Rewards 

° Tasks 

. Leadership 
° Processes 
e Structure 
° People 


The technology, tasks, processes, structure, and people 
factor definitions do not significantly deviate from those 
factors of the same names previously defined in the 
description of the Harvard Business School model. 

Rewards - are defined as the intrinsic and extrinsic 


motivators of people. 
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Leadership - is defined as the art of direct and indirect 
influence and the skill of creating the conditions for 
sustained organizational success to achieve a desired result. 
(DA, EM 22-103, 1987, p. 3) 

The final category considered by the Strategic Leadership 
Planning model is Outputs. As with several factors in the 
above categories, the Strategic Planning model’s definition of 
outputs is similar to that defined in the Harvard Business 
School model. The key difference being that outputs, 
according to the Strategic Leadership Planning model, are 
grouped into three distinct categories: Organizational, 
Group, and Individual. 

Organizational Outputs - are the formal products or services 
produced by the organization. 

Group Outputs - are the products of various groups or 
subsections of the organization, intra-organizational products 
or services. 

Individual Outputs - are those products produced as the result 
of the efforts of a single individual. 

Comparing the Strategic Leadership Planning model to the 
Harvard Business School model, it is clear that the Strategic 
Leadership Planning model more accurately reflects the factors 
addressed by the Harvard model as critical to determine the 
health of an organization. This comparison (Figure 12) shows 
a substantive increase in the areas covered by both models. 
Areas addressed in the Harvard model and covered in the 
Strategic Planning model include the areas addressed by both 


the Harvard model and the Presidential model, plus: 


° Environment 
e Task Requirements 
° Technology 
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Figure 12: A Comparison of the Strategic Leadership 
Planning Model and the Harvard Business School Model. 


° Structure 

° Culture and Management Style 
As with the previous comparison, factors that are similar in 
both models are connected with solid arrows. Factors in the 
Harvard model that are not addressed by the Strategic 
Leadership Planning model are represented with darkened boxes. 
There is one category, outcomes, that is covered by both the 
Harvard model and the Presidential model that is not addressed 
by the Strategic Leadership Planning model. 

Although the Presidential model addressed some of the 
same general categories as the Harvard model, as discussed 
earlier, the definitions and intent of these categories were 
fundamentally different. This is not the case with the 


Strategic Leadership Planning model. As noted above, many of 
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the factor definitions used in the Strategic Leadership 
Planning model are similar in nature and intent to those 
defined by the Harvard model. 

b. The USAREUR Organizational Diagnostic Survey 

The USAREUR Organizational Diagnostic Survey 
(Appendix B) was developed in 1982 by the USAREUR 
Organizational Effectiveness Office as an implementing 
instrument for the Strategic Leadership Planning model. The 
purpose of the survey was to solicit feedback concerning an 
organizations values, organizational design factors, and 
outputs from each member within the organization. The 
original USAREUR Diagnostic Survey consisted of 66 individual 
items, grouped into eight categories aligned with the seven 
organizational design factors and the organizational strategy 
category identified by the Strategic Planning model. The 
survey was modified by this researcher and administered to 
selected members of the CCAWS and AGMS organizations to 
determine if the survey and its supporting model address the 
shortcomings identified with the Presidential Award model and 
the Self-assessment Guide for Organizational Performance and 
Customer Satisfaction and are effective tools to measure the 
program health and customer satisfaction of a project 
management office. The modified survey (Appendix A) 
administered during the prototype assessment testing consists 


of 74 items, grouped as above. 
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V. INSTRUMENT AUTOMATION 


A. INTRODUCTION 

The sponsoring organizations of this thesis requested 
that the assessment instrument recommended by this study 
incorporate a automated approach to the assessment process. 
This chapter describes the candidate software programs 
selected as potential solutions to this requirement. The 
chapter first discusses this researcher’s evaluation of the 
Continuous Improvement (CI) Toolkit to meet the sponsors’ 
specified requirements. The chapter then presents an 
independently developed software program, designed 
specifically for the purpose of administering the assessment 


instrument recommended by this study. 


B. THE CONTINUOUS IMPROVEMENT TOOLKIT FOR WINDOWS 

The Continuous Improvement Toolkit for Windows was 
developed by Systems Improvement, Incorporated in 1984. The 
program was designed as a medium to determine the status of an 
organization, measured against the management’s goals and 
objectives, using an appropriate quality model. It was 
further designed to assist management to implement a plan for 
the correction of identified shortcomings; define, measure, 
and track their root causes; and benchmark the organization’s 
progress against world class companies. (Interviews, 1995) 

After interviewing the developers of the CI Toolkit and 
completing the workshop provided by Systems Improvement, this 
researcher evaluated the CI Toolkit’s appropriateness for use 
in administering the USAREUR Organizational Diagnostic Survey 
and the assessment recommended by this study. The evaluation 


concluded that the CI Toolkit, as designed, does not meet the 
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sponsors’ specific requirements and is not suitable to 
administer the above assessments for the following reasons: 


° The CI Toolkit was not designed to collect feedback 
from 100 percent of an organization. Le Ws 
primarily developed for use by organizations which 
intend to prepare an application for a quality 
achievement award, such as the Baldrige. It 
therefore incorporates a consensus-based approach 
to data collection. Data collected by this process 
are recorded as a single response. 


° Data are stored to a flat, non-relational database 
file. 
® Data recorded by the CI Toolkit are stored ina 


unique format that is incompatible with the 
analysis and database programs currently used by 
the sponsoring organizations. 


° Analyses conducted within the CI Toolkit program 
are fixed and limited to summary statistics. 

° The CI toolkit provides a limited number of 
answering scales. Once chosen, one scale must be 
used to answer all items in the assessment or 
survey. 

° The display screens used to present the assessment 


items are copyrighted and cannot be tailored to 
maximize the administration of the recommended 
survey. 


° The CI Toolkit is not a cost effective solution. 
The sponsors’ require only one of the seven major 
features included in the software program. 


C. PROTOTYPE SOFTWARE DEVELOPMENT 

Based on the evaluation of the CI Toolkit for Windows and 
the sponsors’ specific automation requirements, this 
researcher developed an independent prototype software program 
specifically designed to administer the assessment recommended 
by this study. The prototype consists of two major parts: a 
network-based user interface application for the purpose of 
data collection, and ae data storage, analysis, and 


presentation application. 
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1. User Interface 

The user interface was designed using Novell’s AppWare. 
AppWare, version 1.2, is a commercially available visual 
application builder developed by the Novell Corporation. 
AppWare enables users to develop independent software 
applications using Object Oriented Design techniques. The 
program offers the user a selection of various standard 
subjects, objects, and operators common to the windows 
environment. These subjects, objects, and operators appear to 


the user as icons (Figure 13). 
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Figure 13: AppWare Objects and Operators. 





Fach icon represents a unique set (lines) of code 
required to perform a selected activity. The code, itself, is 
transparent to the user, thus allowing users untrained in 
writing code to develop complex applications. To design a 


program the user simply drafts the desired program hierarchy, 


S51 


structure, and associated logic, places the necessary objects 
and operators within the developed structure, assigns 
parameters for each object and operator, complies’ the 
application, then runs the program. 

The purpose of the user interface is to provide a network 
based, user friendly interface for data collection in a 
windows environment. The program is designed to accomplish 


the following tasks: 


° Instruct users on the assessment procedures. 

° Administer the assessment. 

° Collect respondent feedback to the assessment 
items. 

° Temporarily store the collected data. 

e Export data to a common data format. 

° Initiate the data storage, analysis, and 


presentation program. 
The user interface is designed for installation onto an 
organization’s Local Area Network (LAN). The data flow using 


the LAN is depicted in Figure 14. 
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Figure 14: User Interface Network Diagram. 
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The LAN is the most efficient method to administer an 
assessment within a large or complex organization and provides 
an added degree of flexibility and comfort to both the 
facilitator and participants of the assessment. Data are input 
by each respondent from any available user workstation 
connected to by LAN to the network server. These data are 
sent to the network server and stored in the input database. 
After all data have been entered, the unit facilitator, from 
his/her workstation transfers the data from the input to the 
analysis package. Data and analyses stored in the analysis 
package can be viewed by the unit facilitator or the 
project/product managers from their respective workstations. 

The user interface presents the assessment items using 
the standard Windows graphical environment. Respondents are 
presented a series of screens, each containing an assessment 
item and an appropriate scale to respond to the item. (Figure 
15) 





Totally 


Substantially 


Reasonably 


1.8 To what extent are the Project 
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Figure 15: Example Assessment Item Screen. 
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The respondent simply indicates the appropriate response to 
each item then continues to the next item. Once entered, each 
response is converted to a numerical value and stored in the 
input database. 

2. Data Storage, Analysis, and Presentation 

The data storage, analysis, and presentation application 
was designed using Microsoft’s Assess database program. The 
application is designed to accomplish the following tasks 


using parameters defined in the application development: 


° Import data from the user interface program 

° Format the data 

e Query the data 

° Analyze the data 

e Present analysis results 

° Print the data and analyses 

e Store the data for a five year period 
AGMS 


a 





Question Group #1 
{Male -vs- Female} 

















Average Score 


Figure 16: Example Analysis Presentation. 
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Once imported from the interface database, data are 
automatically formatted and sorted. Queries are 
simultaneously constructed based on the parameters stated 
during the application development. Calculations used to 
analyze the data are completed as part of the querying 
process. Analysis results are presented graphically beginning 
with a macro view of the overall assessment (Figure 16). 
Successive levels of detail can be viewed instantly by 
indicating which category or demographic feature is desired. 
This process is facilitated through the use of macros included 
in the MS Access program and logic statements designed during 


the application development. 


wi 
on 
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VI. ANALYSIS AND DISCUSSION 


A. INTRODUCTION 

The purpose of this chapter is to analyze the data and 
feedback accumulated during the candidate assessment and 
prototype software testing, and to discuss issues concerning 
the applicability and effectiveness of the USAREUR 
Organizational Diagnostic Survey and the prototype self- 
assessment software. The chapter begins with the presentation 
of summarized data and qualitative responses from the surveys 
and interviews conducted by this researcher. These data and 
responses are then used to analyze the appropriateness of the 
individual items contained in the USAREUR Organizational 
Diagnostic Survey (modified) to measure the program health and 
customer satisfaction of a project management office. The 
chapter concludes with a discussion of issues surfaced during 
the prototype testing concerning the design and procedures of 
the USAREUR Organizational Diagnostic Survey (modified) and 


the prototype software application. 


B. DATA 

1. Survey #1 

As previously addressed, the analysis of the data 
collected concerning the program health and customer 
satisfaction of the CCAWS and AGMS project offices is beyond 
the scope of this study and therefore will not be presented 
here. These data were collected to facilitate the candidate 
assessment and prototype software testing and to provide 
realistic data for use in the development of the data storage, 
analysis, and presentation application. Data recorded during 
the prototype testing will, however, be forwarded to the 


sponsoring organizations for their internal use. 


a 


2. Survey #2 

The data for each individual item of survey #2 are 
presented in summarized form in Appendix F. The qualitative 
responses relating to these survey items are listed in 
Appendix G. Assessment items identified as questionable, that 
is revealing a less than 99 percent rating overall and a 
rating of less than 935 percent in one or more aspects, are 


listed in Table 3 below. Deficient aspects are indicated with 
an A 


item | Clarity | Conciseness Scale 











Table 3: Questionable Assessment Items. 


3. Survey #3 
Survey #3 data concerning the candidate self-assessment 
are summarized in Table 4. The qualitative responses related 


to items A4 and A6é are listed in Appendix H. 
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Table 4: Survey #3 Data Summary. 


4. Survey #4 
Survey #4 data concerning the prototype software 
application are summarized in Table 5. The responses to items 


B6 and B7? are listed in Appendix H. 
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Table 5: Survey #4 Data Summary. 
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5. Interviews 
a. Project Manager 
The project manager interview items are in Appendix 
I. Individual responses to the items are listed at Appendix 


J. These responses are summarized as follows: 


(1) The participating project offices intend 


to use the unit self-assessment to: 


® Evaluate the overall health of the organization. 

e Establish a baseline for future comparisons. 

e Pinpoint systemic problem areas within the 
organization. 

. Establish organizational trends. 


(2) The types of analyses most beneficial to 
the project mangers are: 
° Trend analysis by major categories. 


° Comparison of results by demographically 
Significant subsets of the project office. 


e Benchmarking against previous years results and 
against other project offices. 


(3) The types of presentation output most 
beneficial to the project managers are: 


° A slide show presentation, allowing the project 
Manager to view the results of the assessment 
directly from the networked database and print 
selected analytical views. 


° Statistical charts summarizing and comparing 
analyzed data. 

e Bar charts summarizing and comparing analyzed data. 

e Synopses of the qualitative response items. 


(4) The project managers define “program 
health” as: 


e The successful fielding and support of material to 
the soldier. 


° Mission accomplishment. 
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e Personnel job satisfaction. 
. Adequate personnel empowerment and resourcing. 
(5) The project managers feel the unit self- 
assessment is, for the most part, focused on the indicators 
that determine the program health of the organization. The 


assessment needs to include focus on the following areas to be 


comprehensive: 
° Products and services. 
* External influences on the PMO. 
° External and internal customer satisfaction. 
° Integrated product teams. 


(6) The project managers intend to survey 100% 
of their respective organizations using the recommended self- 
assessment instrument. 

(7) The project managers feel the demonstrated 
self-assessment package will provide them with the necessary 
tools to identify weaknesses in their respective organizations 
and monitor their improvement. 

b. Unit Facilitators. 

The unit facilitator interview items are in Appendix 
K. Individual responses to the items are listed at Appendix 
L. These responses are summarized as follows: 

(1) The facilitators expect an automated unit 
assessment to significantly enhance their unit facilitator 


role. They feel the assessment will reduce: 


° Facilitator bias. 
. The burden of analysis. 
® The logistical burden associated with the 


assessment process. 
(2) The facilitators do not expect to 


encounter substantial difficulties as a result of switching to 
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an automated assessment process. The areas in which they 


expect some difficulty are: 


° Tailoring of automated analyses to detect 
demographic differences. 


® Determination of who, by name, has not completed 
the assessment. 


(3) The facilitators prefer to have the data 
analysis and presentation fully automated, with minimal 
facilitator input. Thus, enabling a consistent, pre-defined 
statistical package to be produced for each analysis. 

(4) The facilitators choice of analyses and 


output match those of their respective project managers. 


C. ANALYSIS 


This section presents the analysis of the data and 
qualitative responses summarized above. The analysis focuses 
on the two major components of the proposed unit self- 
assessment package: The USAREUR Organizational Diagnostic 
Survey and the prototype software application. 


1. The USAREUR Organizational Diagnostic Survey 
(modified) 


The analysis of the USAREUR Organizational Diagnostic 
Survey is divided into four areas: Questionable assessment 
items; Instructions and procedures; Length and time required; 
and applicability to a project management office. 

a. Questionable Assessment Items 

Item 1.3. To what extent are the objectives of the 
Project Office realistic? 

This item was rated as irrelevant. Participants 
rating this item as irrelevant can be grouped into two 
categories. Several participants stated that due to their 
position in the organization they were unfamiliar with the 


project office’s objectives, thus they had no basis from which 
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to judge their realism. These participants generally felt 
that the objectives of the project office and this item do not 
apply to persons in their positions. The second category of 
participants rating this item as irrelevant stated that the 
objectives of the project office are, in effect, determined by 
management personnel and other entities external to the 
project office itself and therefore should not be addressed by 
an internal assessment of the project office. 

Although it is imperative that all personnel within 
a given organization be familiar with that organization’s 
objectives, as measured by item 1.2, it is clear that not all 
of these personnel are in a position to be able to determine 
the realism of these objectives. It is possible, however, for 
all personnel within a project office to judge the realism of 
the objectives of their individual working group, branch, or 
division, given the group’s present resources and the 
objectives of the next higher organization. To this end, 
objectives for mission accomplishment should be formulated at 
every level within the organization. It is for this reason 
that the issue raised by the second group of participants 
above is invalid. This item’s focus is on those objectives 
for mission accomplishment determined by the management within 
the project office. The survey item is, however, too general 
in scope to allow all personnel within the project office to 
respond accurately without speculation. Since the purpose of 
this survey is to elicit feedback from the entire 
organization, this item, as written, is irrelevant. 

Item 1.6. I am satisfied with my role in setting 
organizational objectives. 

This item was rated as irrelevant. Participants rating 


this item as irrelevant stated that persons of their position, 


as well as many other positions in the organization, do not 
influence the development of organizational objectives. 

Setting direction within an organization, to include 
establishing organizational objectives, is a function of 
management. Therefore, a majority of the individuals within 
a project office would not be expected to play a role in 
accomplishing this task. Since this survey is designed to 
apply to all personnel within the project office, this item is 
irrelevant. 

Item 1.7. To what extent are the Project Office's 
objectives for mission accomplishment consistent with the 
demands of its environment? 

This item was rated as unclear, wordy, ambiguous, 
and irrelevant. Participants rating this item as unclear, 
wordy, and ambiguous stated they did not understand the phrase 
“the demands of its environment” and therefore also did not 
understand what type of response was being solicited. 
Participants indicating the item was irrelevant stated the 
“objectives for mission accomplishment” did not pertain to 
person in their position within the organization. 

The phrase “the demands of its environment” implies 
a potentially infinite number of variables. This phrase also 
presupposes that all personnel within the organization are 
familiar with the organization’s environment and the demands 
placed on the organization by external entities. Thus, this 
item, as written, lacks sufficient definition to solicit a 
singular response and therefore is both unclear and ambiguous. 
As with item 1.3, this item requires the majority of personnel 
within the project office to speculate as to the 
appropriateness of the project office’s objectives. AS 
discussed above, many persons within the organization are not 


in a position to respond to items of this type with any 


significant degree of accuracy, thus this question is also 
irrelevant. 

Item 1.8. To what extent are the Project Office's 
objectives for mission accomplishment consistent with the 
availability of its resources? 

This item was rated as unclear, wordy, and 
ambiguous. Participants rating the item in this manner stated 
they did not understand the phrase “the availability of its 
resources” and therefore also did not understand what an 
appropriate response should be. 

This item, as in the case of item 1.7, presupposes 
knowledge of the project office’s available resources and 
requires a stricter definition of the term resources to enable 
personnel within the organization to respond accurately. 
Thus, this item, as written, is unclear, wordy, and ambiguous. 

Item 1.10. The strategy of the Project Office could 
be improved by: 

This item was rated as unclear and ambiguous. 
Participants who indicated that this item was unclear or 
ambiguous were troubled by the word “Strategy”. They either 
did not fully understand what an organizational strategy was 
or they were not familiar with the strategy of their 
respective organization. 

This rating is understandable, given that this 
survey is intended for all members of the organization. 
Strategy development is clearly a function of management. 
Personnel other than management are therefore not expected to 
be in a position to comment on strategy improvement. All 
personnel, however, should be knowledgeable as to what the 


organization’s strategies are. This item, although clear and 


unambiguous, is inappropriate for this survey. 





Item 2.2. To what extent do your assigned tasks 
include work that you consider to be outside your area of 
responsibility? ; 

THis item and its answer scale were rated as 
inappropriate. Participants rating this item’s answer scale 
as inappropriate stated the scale did not provide appropriate 
responses to answer the item. They suggested the item be 
reworded as a statement, using the Agree/Disagree scale. More 
importantly, this item was viewed by several leaders within 
the organization as a source of potential grievances due to 
its wording. For this reason alone the item is inappropriate. 

Item 2.3. To what extent are your assigned task 
deadlines reasonable? 

This item was rated as unclear. Participants rating this 
item as unclear felt the item lacked sufficient definition. 
One group of participants stated the item should be focused on 
normal or routine tasks, eliminating those tasks associated 
with the natural level of inherent crisis management 
associated with the project management discipline. Another 
group of respondents stated the word “reasonable” was not 
adequately defined. 

Although there is a certain amount of crisis 
management involved in the normal operation of all 
organizations, there is not sufficient rationale to discount 
this fact in this items wording. In fact, it is this level of 
crisis management that the item is attempting to measure. To 
what extent does the management of the project office rely on 
crisis management techniques, rather than proactive planning? 
Second, “reasonable” is inherently defined as the participants 
opinion as to what is reasonable and therefore it would be 
constraining the survey participants by further defining this 


term. This item is clear as written. 
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Item 3.1. To what extent does the Project Office's 
present structure facilitate work? 

This item was rated as unclear. The participants 
indicating this item was unclear stated that the word “work” 
was not adequately defined. 

This researcher agrees that the word “work” in this 
context appears unbounded. This item should be rewritten to 
focus on the extent the project office’s present structure 
facilitates or hinders the participants ability to accomplish 
his/her assigned tasks. This item, as written, is unclear. 

Item 3.3. To what extent does the Project Office's 
present structure facilitate decision making? 

This item was rated as ambiguous. Several 
participants indicated this item was ambiguous stating that 
the level of decision making being considered needs to be 
defined. 

This researcher agrees that the level of decisions 
being considered is an important element to be included in 
this item. This item should focus at a level at which all 
personnel within the organization can respond to the item 
without speculation. This item, as written, is ambiguous. 

Item 3.4. Our informal work structure complements 
the established formal structure. 

This item was rated as unclear. Participants 
indicating this item was confusing did not understand the 
references to the “informal work structure” and “formal 
structure”. 

It is apparent to this researcher how these terms 
could be a source of confusion, given the structure and 
environment of the military project management office. Within 


the overall structure of the project office there are multiple 


formal and informal sub-structures. First, there is the 





traditional line authority structure of the project office. 
This structure does not, however, include all personnel 
dedicated to the project. Many of the personnel directly 
supporting the project office are matrix personnel. That is, 
they are assets belonging to other organizations that are 
dedicated to and work within the project office. Each of 
these individuals also reports to superiors in their parent 
organization. Second, much of the work in the project office 
is accomplished within Integrated Product Teams (IPT). These 
IPTs are informal structures developed to facilitate the 
efficient interaction between all functional areas involved in 
the production of a particular product. Finally, there is 
usually an ad hoc structure developed by the workers in an 
organization that is used, as necessary, in lieu of the stated 
formal structure to accomplish required tasks in the most 
efficient manner. Depending on the position and parent 
organization of the survey participant it could be unclear to 
which formal or informal structure this item is referring. 
This item, as written, is unclear. 

Item 3iad< To what extent are individual 
responsibilities defined in the Project Office? 

This item was rated as unclear and ambiguous. 
Personnel indicating this item was confusing felt this item 
was too general and asked for information that cannot be 
obtained from personal knowledge. 

This researcher agrees that this item solicits a 
speculated response. This item should be worded to refer only 
to the participant’s responsibilities. This item, as written 
is unclear and ambiguous. 

Item 6.3. To what extent is conflict dealt with 


openly and managed constructively in the Project Office? 


This item was rated as unclear. Participants rating 
this item as unclear stated that the item was confusing 
because the types and level of conflict are undefined. 

This researcher agrees that the types and levels of 
conflict being measured are critical components of the item. 
Within any organization there is a broad range of conflict 
types, such as personal conflicts, conflicting tasks, 
conflicts between superiors and subordinates, etc. The type 
of conflict and the level at which it occurs in the 
organization must be clearly defined to enable the survey 
participant to accurately respond to this item. This item, as 
written, is unclear. 

b. Instructions and Procedures 

The instructions provided on how to complete the 
assessment were relatively complete and clear to most of the 
survey participants. Of the participants indicating the 
instructions were incomplete, most referred to a lack of 
guidance in the use of the provided software and not the 
survey instrument itself. This is a separate issue discussed 
in the analysis of the prototype software. Those participants 
stating the instructions were confusing indicated that the 
reference to the types of items included in the survey were 
confusing. The survey instructions refer to items soliciting 
a qualitative response as “free” or “open” response items and 
items requiring the participant to select an appropriate 
response from a fixed set of responses as “fixed” response 
items. This shortcoming is easily reconciled by using more 
traditional terms, such as “short answer” and “multiple 
choice” to refer to the item types. Once the instructions 
were clarified, none of the participants encountered any 
significant difficulties associated with the survey 


procedures. Again, those difficulties that did surface were 
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connected to the use of the software and were not linked to 
the assessment instrument. 

c. Length and Time Required 

The candidate survey consisted of 74 items, compared 
to 66 items incorporated by the Self-assessment Guide for 
Organizational Performance and Customer Satisfaction. The 
majority of the survey participants indicated that the length 
of the survey was adequate to ensure a comprehensive 
examination of the organization’s program health and customer 
satisfaction without requiring an excessive amount of time to 
complete. Several participants did indicate that they felt 
the survey contained too many items, however the time required 
to complete the survey was acceptable. 

The average time required for each participant to 
complete the survey was 44 minutes. The 44 minute average time 
to complete the survey is inflated due to each participant 
providing feedback on the survey items as they completed the 
survey. This researcher estimates that an average of ten 
minutes of the time required for each participant to complete 
the survey was dedicated to providing this feedback. It is 
therefore estimated that the required time to complete this 
survey, on average, is 34 minutes. This is compared to an 
estimated time of 930 minutes necessary to complete the Self- 
assessment Guide for Organizational Performance and Customer 
Satisfaction. 

Although the candidate survey contains eight items 
more than the Self-assessment Guide for Organizational 
Performance and Customer Satisfaction, it is significantly 
shorter and requires less time to complete than its 
predecessor. There are two reasons supporting this 
conclusion. First, although the Self-assessment Guide for 


Organizational Performance and Customer Satisfaction contains 
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only 66 items, the responses provided for these items are 
lengthy sentences or, in some cases, paragraphs. This 
requires the participant to dedicate a substantial amount of 
his/her time to read each response prior to answering the 
items. Highlighting this is the fact that the 66 items of the 
Self-assessment Guide for Organizational Performance and 
Customer Satisfaction span 44 pages of text, whereas the 74 
items of the candidate assessment are covered in six pages. 
Second, the candidate assessment incorporates an automated 
approach, allowing the participant to quickly view the item, 
indicate his/her response and continue to the next iten, 
whereas the Self-assessment Guide for Organizational 
Performance and Customer Satisfaction does not. 
d. Applicability to a Project Management Office 
The majority of the survey participants indicated 
that the candidate assessment, as a whole, is applicable to a 
project management office. These participants felt, however, 
that many of the survey items were not appropriate for persons 
of their position within the organization. The main reason 
cited for this observation was that the scope of these items 
were too broad and included elements outside of their purview. 
This researcher whole heartedly agrees with this observation. 
In fact, this observation applies to the majority of the items 
included in the assessment. This observation is addressed in 
detail in the discussion section of this chapter. 
2. Prototype Software Application 
The analysis of the prototype software application is 


divided into two areas: Instructions and procedures; and Fase 





of use and time required. 
a. Instructions and Procedures 
The majority of the survey participants indicated 


that the software instructions were comprehensive and clearly 
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stated. The only difficulty associated with the instructions 
involved identifying particular items as they actually appear 
within the software, based on their description in the 
instructions. This shortcoming is addressed in the first 
production version of the software by including the 
instruction within the application’s on-line help and 
providing graphical examples of the items, screens, and other 
objects unique to the software application. 

b. Ease of Use and Time Required 

All of the survey participants, without exception, 
stated that the software application was “user friendly” and 
that they encountered only minor difficulties completing the 
survey using the application. All of the participants, except 
one, indicated that the Microsoft Windows environment was 
installed on their workstation and therefore the majority of 
the procedures and items incorporated into the software 
application were already familiar to them. The participants 
stated that the automating of the assessment significantly 
enhanced the overall assessment process, requiring them less 
time to complete the assessment than a manual system and 


providing them with increased flexibility and comfort. 


D. DISCUSSION 

The discussion of the issues related to the proposed unit 
self-assessment package, as in the analysis section, is 
focused on the packages two major components: The USAREUR 
Organizational Diagnostic Survey and the prototype software 


application. 


1. The USAREUR Organizational Diagnostic Survey 
(modified) 


a. Assessment Items 
In addition to the items identified as questionable 


in the analysis section of this chapter, numerous other items 
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in the candidate assessment contain shortcomings identical to 
those identified with these items. These items fall into 


three categories: 


° Items to which all participants cannot accurately 
respond without significant speculation because the 
item is focused at too high a level within the 
organization (i.e. project office rather than 
section) or the scope is unbounded. 


e Items which requires participants to infer the 
opinions of other people within the organization. 

° Items in which the subject matter limits meaningful 
responses to a select group of people within the 
organization. 


The first category, items focused at too high a 
level within the organization, applies to a significant number 
of the items included in the candidate survey. Items within 
this category solicit generalized responses from participants 
that require them to draw conclusions about the entire 
organization. These items should be focused at the lowest 
organized level within the organization to allow participants 
to provide accurate responses concerning these items from a 
individual perspective, without speculation. Trends and 
conclusions concerning the entire organization are then 
revealed by combining these individual responses into an 
aggregate information database. Items included in this 


category are: 





Item 1.2 Item 4.2 Item 6.8 Item 7.7 
Item 1.9 Item 4.5 Item 6.9 Item 7.8 
Item 2.8 Item 4.6 Item 7.1 Item 8.1 
Item 2.9 Item 4.7 Item 7.2 Item 8.3 
Item 3.1 Item 6.2 Item 7.3 Item 8.5 
Item 3.10 Item 6.3 Item 7.4 Item 9.1 
Item 3.11 Item 6.5 Item 7.5 Item 9.2 
Item 4.1 Item 6.6 Item 7.6 Item 9.3 


The CCAWS/AGMS Unit Self-assessment reflects adjustments made 
to these items with regard to the organizational level at 


which these items are focused. 
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The second category, items that require participants 


to infer the opinions of other people, includes: 


Item 1.5 Item 3.8 Item 5.4 Item 6.4 
Item 3.5 Item 4.3 Item 5.6 Item 6.7 
Item 3.6 Item 4.4 Item 6.1 Item 8.4 
Item 3.7 


These items have also been adjusted in the CCAWS/AGMS Unit 
Self-assessment to permit participants to respond to each item 
based on individual knowledge. 

The final category above, items whose subject matter 


limit accurate responses to a select group of people, 
includes: 


Item 1.3 Item 1.7 Item 3.3 Item 8.2 
Item 1.4 Item 1.8 Item 5.1 Item 8.7 
Item 1.6 Item 1.10 


These items do not meet the intent of the survey design and 
have been omitted from the CCAWS/AGMS Unit Self-assessment. 

b. Functional Areas and Model 

The USAREUR Organizational Diagnostic Survey 
(modified) evaluated during the prototype testing consisted of 
eight functional areas, described in Chapter IV, and a general 
category. Based on the qualitative responses gathered during 
the evaluation of the candidate assessment and from interviews 
conducted with the project and deputy project managers of the 
sponsoring organizations, it is evident that the candidate 
assessment, although a more appropriate solution than the 
Self-assessment Guide for Organizational Performance and 
Customer Satisfaction, does not effectively address all of the 
critical factors that determine the program health and 
customer satisfaction of a project management office. 
Specifically, the candidate assessment addresses two 
functional areas determined by the project managers to be 


inconsequential to the program health and customer 
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satisfaction of a project office and fails to address three 
additional areas, cited by these managers as critical factors. 

The two areas recommended for deletion from the 
assessment were technology and rewards. Technology was said 
to be inconsequential because the sponsoring project offices 
are fully automated and there is no apparent weaknesses in 
this area. This researcher disagrees with this rationale for 
the following reasons. First, although each of the sponsoring 
project offices may be considered fully automated and void of 
shortcomings in this area at present, this assessment is 
designed to be conducted annually for an undetermined period. 
Given the speed at which technological advancements are 
forthcoming, the technological state at which these two 
organizations are assessed could significantly deteriorate 
from one period to the next. Second, the context in which 
technology is used by these managers and in the candidate 
assessment refers only to the equipment available to the 
project offices. In reality, technology encompasses not only 
this physical equipment, but also the basic methods and 
techniques used to maximize equipment usage. Therefore, the 
scope of this functional area should be broadened and the 
items within this area reviewed to reflect this definition of 
technology. 

The rewards item group was recommended for deletion 
because rigid policies directed by authorities external to the 
project office limit the latitude of managers within the 
project office to effect change in this area. Although this 
is true, in part, leaders within these organizations retain 
the authority to ensure rewards are equitably distributed 
within these guidelines. These guidelines do, however, 
minimize this areas effect to attribute to the overall program 


health of the project office. Therefore, the critical items 
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remaining in this area can be merged into the people category 
and this area deleted from the assessment. 

The factors critical to the program health and 
customer satisfaction of a project management office that were 
not addressed by the candidate assessment are: Outputs (i.e. 
products and services), the Integrated Product Team (IPT), and 
Customer Satisfaction. 

Given the addition and deletion of the areas 
described above, a new model was developed depicting the 
factors critical to the assessment of program health and 


customer satisfaction of a project management office (Figure 
ule 
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Figure 17: The CCAWS/AGMS Unit Self-assessment Model. 
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This model accurately depicts the inputs associated with the 
military project management environment and reflects the 
increased importance of the organizational vision and the 
incorporation of multi-organizational design factors used in 
the Integrated Product Team. 

Comparing this model to that of the Harvard Business 
School, it is clear that in addition to accurately reflecting 
the critical factors that determine program health and 
customer satisfaction in a project office, the CCAWS/AGMS Unit 
Self-assessment model sufficiently addresses the 
organizational design factors identified as critical by the 


Harvard School of Business (Figure 18). 
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Figure 18: A Comparison of the CCAWS/AGMS Unit Self- 
assessment Model and the Harvard Business School Model. 
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c. Program Health and Customer Satisfaction 

The candidate assessment instrument, in its final 
form, is appropriate and effective to measure the program 
health and internal customer satisfaction of a project 
management office. This instrument, however, does not 
sufficiently measure the satisfaction of customers external to 
the project office itself. 

2. Prototype Software Application 

a. Computer Resources 

The prototype software application is compatible 
with the current computer resources used in the CCAWS and AGMS 
project offices. Both project offices are running 
International Business Machines (IBM) or compatible 
workstations. Each of these workstations has the equivalent 
of a 386DX-25 or faster Central Processing Unit (CPU) with a 
minimum of four megabytes of Random Access Memory (RAM). The 
workstations are operated by a Microsoft Disk Operating System 
(MS-DOS), version 5.0 or greater, in the MS Windows 3.1 or 
3.11 environment. Monitors at these workstations contain 
video cards capable of supporting resolutions of 640x480 or 
800x600 pixels. The minimum workstation requirements for use 
with the user interface application are: CPU, 386SX-20; RAM, 
MB; MS-DOS, version 5.0; MS Windows, version 3.1; video 
resolution, 640x480. Each of the sponsoring organizations use 
a Local Area Network (LAN) manufactured by the Novell 
corporation. The user interface is designed to operate ona 
Novell LAN, however, compatibility tests with the LANs in the 
CCAWS and AGMS project offices was not conducted as part of 
the prototype software evaluation. 

The data storage, analysis, and presentation 


application is designed to operate within Microsoft Access, 


version 6.0. This program is currently being used by both the 
CCAWS and AGMS project offices. 

b. User Interface 

The user interface application performed remarkably 
well during the prototype testing. The application compiled 
data from 50 respondents with no significant failures. There 
were, however, several minor shortcomings associated with the 
application that are characterized as normal occurrences 
during the beta testing of a new software application. These 
shortcomings and their solutions are discussed below. 

(1) Method of Indicating Responses. At the 
time of testing, the user interface provided each respondent 
with two methods to indicate their response to fixed 
assessment items. The respondents could either drag the 
indicator up and down the answer scale using his/her pointing 
device (mouse) or perform the same maneuver using the arrow 
keys on the keyboard. During testing this researcher 
observed, however, that the first inclination of each 
respondent, with very few exceptions, was to attempt to 
indicate his/her response by pointing at the desired response 
itself with the pointing device. This was predominately the 
case, even though each respondent had read the software 
instructions detailing the two included procedures. As 
respondents attempted to indicate their responses in this 
fashion, to which the application did not respond, each became 
confused and either referred to the software instructions or 
prompted the researcher for assistance. Since one of the 
primary goals in developing the user interface application was 
to simplify the assessment process by providing a “user 
friendly” interface for data collection, this researcher 


modified the interface application to enable respondents to 
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indicate their responses by pointing at the desired response, 





as was their propensity to do so. 

(2) File Structure. The user interface 
application at the time of testing consisted of ten separate 
executable software programs, a database control and nine 
functional areas, and their associated Dynamic Link Library 
(DLL) files. Although this arrangement simplified the 
packaging and transportation of the application, it created 
considerable problems in the functioning of the application. 
First, because each functional area was contained in a 
separate executable file, each time a respondent entered into 
a new functional area (e.g. Strategy to Task) a significant 
pause was encountered, approximately five seconds, as one 
executable program was opened and the other closed. These 
delays created considerable confusion with each respondent. 

Second, since the database control function was 
located in a separate executable file than each functional 
area, difficulties were encountered identifying the correct 
database record within each functional area that data were to 
be written to or stored in. At the time of testing this was 
accomplished through the use of an external file, created by 
the database function, containing the respondent 
identification number. Each functional area, once opened, 
queried the contents of this file to determine the record in 
which the data were to be stored. Although this technique 
performed successfully during the prototype testing, the 
testing was conducted on a stand alone system. Once the 
application is loaded onto the organization’s Local Area 
Network (LAN) this will not be the case. When the application 
is used on a LAN there is a high probability that two or more 
respondents will access the application at the same time. If 


this situation occurs, the first respondent to complete 
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entering his/her demographic data and begin the assessment 
will have his/her respondent number recorded to the external 
file. Again, this file is used by successive programs to 
determine his/her record in the database. Once the second 
person begins the assessment, however, the application will 
record his/her respondent identification number to the 
external file, overwriting the first person’s data. After 
that time when the first person moves from one functional area 
to another, the external file will provide the current or 
updated information back to the application, thus data entered 
by the first person would be inadvertently stored in the 
second person’s database record, destroying both sets of data. 

Each of the above surfaced anomalies were 
corrected by restructuring the application’s program files. 
The application’s first production version consists of one 
executable file, containing both the database control and all 
functional areas, and its associated DLL files. This 
structure allows a seamless transition from one functional 
area to the next and enables the database function to 
accurately track the respondents record identification number 
throughout the assessment process. 

(3) Exporting Data. One of the sponsor’s 
automation requirements was that the recommended software 
application be compatible with the existing computer resources 
available within their organizations. Since the database 
function included in Novell’s AppWare development program 
stores data to a unique database file, it was essential to 
export data from this file to a common or standard format. Of 
the standard data formats available, the two most common are 
the comma delimited and the tab delimited. The comma 


delimited format uses commas to separate data elements within 


each data record. The tab delimited performs the same 





operation using tabs. Data stored in either of these two 
formats can be readily imported into Microsoft Access and 
Excel, used by the sponsoring organizations, or a number of 
other popular database, spreadsheet, and analysis programs. 
Of these two formats this researcher chose the comma delimited 
format for use in the prototype application. During testing, 
however, it was noticed that many of the respondents preferred 
the use of commas to clarify statements when answering the 
open response items. When exporting these data into a comma 
delimited data format, the application interprets the commas 
internal to each open response item as the beginning of a new 
response (i.e. a response including three commas is 
interpreted as four responses). Therefore, when the data are 
imported into the storage, analysis, and presentation 
application, all responses following an open response with 
internal commas are misaligned by the number of commas 
included in the defective response. This occurrence would 
corrupt the data included in that particular record and 
invalidate any analyses performed on the entire set of data. 

This anomaly has also been corrected in the 
first production version of the software application. The 
production version incorporates the tab delimited data format 
to export data for use in the data storage, analysis, and 
presentation application. This format will enable respondents 
to use any and all characters in their open item responses 
without interfering with the data export function. 

(4) Next/Previous Item Function. At the time 
of testing, the user interface application incorporated push 
buttons to enable participants to move from one assessment 
item to the following item. During testing, a majority of the 
participants revealed a strong desire to return to items 


answered previously to review or compare their answers. Te. 


82 


response to this feedback, push buttons have been inserted 
into each item screen to allow participants to move both 
forwards and back through the available items. 

(5) Resume Function. The user interface 
application incorporates a resume function that allows the 
participant to discontinue the assessment and finish the same 
assessment at a later time. At the time of testing, this 
function required that the participant had written down or 
remembered his/her respondent identification number, assigned 
at the beginning of the assessment. This function was also 
limited in the fact it would only allow participants to return 
to the first item in the desired functional area. During 
testing, it was revealed that, without prompting by the 
researcher, participants did not understand the significance 
of remembering their respondent identification number until 
after they attempted to resume the assessment. The production 
version of the software application addresses this shortcoming 
by including a screen that appears to the participant when 
he/she attempts to terminate the assessment process at any 
point other than the final item. This screen will provide the 
participant with his/her respondent identification number and 
the number of the last item completed and prompt the 
participant to record this information for later use. 

(6) Demographic Lists. The user interface 
application queries each participant for demographic and other 
essential information using a set of fixed “drop down” lists. 
This information is used by the analysis and presentation 
application to group data for analytical purposes. Listings 
used in the application were compiled using information 
provided by the CCAWS and AGMS project offices. The 
categories represented are: Year; Organization; Office; 


position; time in organization, office, position; Age; Sex; 
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Military/Civilian; Rank/Grade; and Education level. Despite 
the attempt to compile a comprehensive listing of the 
applicable items within each of these categories, it was 
revealed during the prototype testing that some of the lists 
were incomplete. An attempt at a one time fix to this problem 
would most probably also end in failure. Therefore, the 
production version of the user interface application 
incorporates a function that will enable the unit facilitator 
to add items to each of these lists during “run time”, as 
necessary. 

c. Data Storage, Analysis, and Presentation 

The data storage, analysis, and presentation 
application was not available during the prototype software 
testing in June 1995. This application was, however, 
demonstrated for the CCAWS Project Manager in August 1995. 
Results of this demonstration and comments from the unit 
facilitators and the deputy project managers from both 
organizations as to the design of this application are 
discussed below. 

(1) Data Storage. The sponsors’ requirements 
to automate the assessment process included a data storage 
function capable of storing collected data for a period of not 
less than five years. Both the user interface and the data 
storage, analysis, and presentation applications meet this 
requirement. The user interface, although designed for 
temporary storage of data during the assessment process, is 
limited only by the number of years recorded in the year list 
within the application. Since this list can be updated by the 
unit facilitator, as necessary, the capacity of the user 
interface to store data for the purposes of the unit self- 
assessment is unlimited. The application, as delivered, 


provides an initial capability to store 12 years of data. The 
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data storage, analysis, and presentation application is 
designed for the permanent storage of collected data. Its 
capacity to accomplish this task, as delivered, is unlimited. 
It is important to note, however, that data modified directly 
from within the data storage, analysis, and presentation 
application after they are initially stored must be saved to 
a separate file. Although these data can still be used, once 
data from the next assessment iteration are imported from the 
user interface, these data will be overwritten and all changes 
to the data made in the data storage, analysis, and 
presentation application will be lost. This function was 
purposely incorporated to purge the database of duplicate 
records. The function does not compare data records to locate 
duplicates, rather it keys on the dates the data were entered. 
Each data record is date stamped as it is imported into the 
permanent database. As new data are imported, the application 
purges all records with a date stamp that is earlier than the 
import date. This process does not permanently erase previous 
years data, as they were originally entered, because these 
data are again imported into the data storage, analysis, and 
presentation application along with the new data records. 

(2) Analysis. The data storage, analysis, and 
presentation application is designed to automatically format, 
store, query, and perform calculations on data as they are 
imported into the permanent database. Calculations performed 
on the data by the analysis function, as delivered, meet the 
requirements of the CCAWS and AGMS Project Managers. These 
calculations, however, were designed from within the Microsoft 
Access database program and can be altered to perform 
additional analyses, as necessary. The analyses currently 


performed by the analysis function are: 


e The percentage of actual responses of each 
functional category, by organization and year, to 
the maximum possible response. 


° The average response for each assessment item, by 
year, organization, office, and demographic 
category. 

° Groups qualitative responses by year, organization, 


office, and demographic category. 
These calculations are then compared to and against each other 
to provide the manager with an in-depth analysis of each item 
measured by the self-assessment. 

(3) Presentation. The presentation function of 
the data storage, analysis, and presentation application is 
designed to graphically present analyses performed by the 
analysis function in a manner that provides useful feedback 
and information to the project office’s management. At the 
time of testing, this function was designed to print a 
“canned” or fixed set of graphs, tables, and charts to be used 
by management personnel. During the demonstration of the data 
storage, analysis, and presentation application, however, the 
CCAWS Project Manager commented that the presentation function 
would be more useful if it allowed managers to view these 
graphs, tables, and chart at their workstation and print only 
those items that they deemed significant. The production 
version of the data storage, analysis, and presentation 
application incorporates this suggestion, allowing each 
presentation item to be viewed on screen and printed, as 
necessary. The presentation graphics are structured in a 
hierarchical fashion. As the database application is opened, 
the manager is presented with an overall assessment of the 
organization depicting each year the assessment instrument has 
been used. The manager is provided a series of push buttons 
in the screens margins allowing him/her to choose the year 


he/she desired to examine. Once a year has been selected, the 
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graph depicting that year by functional category is presented 
to the manager. This procedure is then repeated, each level 
providing increased detail, until completed. Presentation 
graphics depicting six levels of detail are included in the 
application (i.e. Item 1.1 by year, organization, office, 


position, and demographic category) as delivered. 
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VII. CONCLUSIONS AND RECOMMENDATIONS 


A. INTRODUCTION 

The purpose of this chapter is to draw conclusions 
concerning the effectiveness of the CCAWS/AGMS Unit Self- 
assessment and the prototype software application to measure 
the program health and customer satisfaction of a military 
project management office, based on the issues and findings 
revealed in the data analysis and discussion. The chapter is 
divided into five sections: Conclusions drawn from the 
analysis and discussion; recommendations for implementing the 
CCAWS Unit Self-assessment and the prototype software 
application; thesis contributions; research limitations 
discovered during the data analysis; and recommendations for 


further study. 


B. CONCLUSIONS 
This study addresses the primary research question: 


Does the proposed unit self-assessment package satisfy the 
sponsors' requirement to efficiently and effectively assess 
and track program health and customer satisfaction? 


The proposed unit self-assessment package, consisting of 
the CCAWS/AGMS Unit Self-assessment and the Unit Self- 
assessment software, version 1, do provide the military 
project manager with the tools necessary to efficiently and 
effectively assess the program health and internal customer 
satisfaction of a project management office. The CCAWS/AGMS 
Unit Self-assessment does not, however, satisfy the sponsors’ 
requirement to assess the external customers’ satisfaction 
with the products and services provided by the project office. 


This thesis also sufficiently addresses each subsidiary 


question listed below: 





1. Does the proposed assessment address and correct the 
deficiencies of the Self-assessment Guide for Organ- 
izational Performance and Customer Satisfaction, 


identified by the sponsors’ management team and 
users? 


The CCAWS/AGMS Unit Self-assessment provides enhanced 
solutions to each of the problems identified by the management 
teams and personnel from the CCAWS and AGMS Project Management 
Offices. Specifically, the CCAWS/AGMS Unit Self-assessment 
addresses improvements in the following areas: 

a. Applicability of Assessment Items 

All of the items encompassed within the CCAWS/AGMS 
Unit Self-assessment are applicable to a military project 
management office. Items determined as questionable by CCAWS 
and AGMS personnel during the candidate assessment evaluation 
were either modified to fit the project management environment 
or discarded. 

b. Item Wording and Content 

The items in the CCAWS/AGMS Unit Self-assessment are 
worded in a manner that is both clear and understandable by 
all members of a military project office. Questionable items 
during the candidate assessment evaluation in this regard were 
either simplified or reworded entirely to ensure each 
participants full understanding. 

c. Assessment Length 

The CCAWS/AGMS Unit Self-assessment’s length is 
sufficient to ensure a comprehensive evaluation of the program 
health and internal customer satisfaction of a military 
project management office, without requiring an excessive 
amount of time for each participant to complete. The average 
time for each participant to complete the candidate assessment 


will be 34 minutes, as compared to an estimated 90 minutes to 
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complete the previous Self-assessment Guide for Organizational 
Performance and Customer Satisfaction. 

d. Target Audience 

The items within CCAWS/AGMS Unit Self-assessment are 
appropriate to solicit an accurate response from all personnel 
within a military project management office. Items in which 
the subject matter or organizational level restricted the 
ability of all personnel to provide an accurate response, 
based on their personal knowledge were either reduced in scope 
to enable all personnel to respond or eliminated from the 
assessment. 

e. Qualitative Feedback 

The CCAWS/AGMS Unit Self-assessment enables each 
participant to provide both quantitative and qualitative 
feedback in each evaluated category, as opposed to the Self- 
assessment Guide for Organizational Performance and Customer 
Satisfaction, which incorporates qualitative feedback by means 
of a consensus-based management approach. 


2. Is the proposed software program an effective method 
to administer the proposed assessment? 


The independently developed prototype software 
application is an effective method of automating the unit 
self-assessment process and administering the CCAWS/AGMS Unit 
Self-assessment. The software has proven effective in 
reducing the overall time associated with the assessment 
process by simplifying the activities of both the assessment 
participants and the designated unit facilitators. The 
average time for each participant to complete the candidate 
assessment using the prototype software will be 34 minutes, as 
compared to an estimated 90 minutes to complete the Self- 
assessment Guide for Organizational Performance and Customer 
Satisfaction using manual procedures. This amounts to a 


savings of 93% man-hours, in a project office of 100 assigned 
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personnel, per assessment iteration. The software 
application’s analysis and presentation function 
instantaneously performs the calculations required to analyze 
the collected data, simultaneously formatting the analyzed 
data into statistically relevant charts and graphs for 
presentation of the analysis. This procedure, accomplished 
manually, is estimated to take one person a full eight hour 
work day to complete. Thus, the prototype software 
application is estimated to reduce the overall assessment 
process time by 506% man-hours, over a five year period. 

The analysis of the prototype software application 
also addresses the following sponsor automation requirements: 

a. Compatibility with Existing Computer Resources 

The prototype software application is fully 
compatible with the current hardware and software 
configurations within the CCAWS and AGMS Project Offices. The 
Minimum system specifications of the LANs and computer 
workstations in each of the project offices are sufficient to 
enable the installation and operation of the prototype 
software application. Programs within the prototype 
application are specifically designed to operate from within, 
or interact with, Microsoft’s Access database program, which 
is used by both the CCAWS and AGMS Project Offices. 

b. User Friendly 

Based on the responses from the CCAWS and AGMS 
participants in the prototype software testing, this 
researcher concludes that the application is “user friendly”. 
During this testing, none of the participants encountered any 
significant difficulties related to the use of the software 
application. Most of the participants stated that the 
software environment, and many of the objects and operators 


were already familiar to them. This is credited to the fact 
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that the software incorporates the use of the Microsoft 
Windows environment, used in existing applications within both 
the project offices. For those objects that are unique to the 
prototype application, participants of the prototype testing 
were observed during the evaluation process to determine their 
initial reactions to these items. The data collected from 
these observations were then used to adjust the initial 
prototype, aligning the commands activating the application’s 
functions to correspond with the participants’ inclinations to 
activate them, thus further enhancing the “user friendliness” 
of the application. 

c. Accessibility 

The CCAWS/AGMS Unit Self-assessment is easily 
accessed by all members of the CCAWS and AGMS project offices. 
The assessment software, once installed, is located on the 
organization’s LAN. Each workstation within the organization 
connected to this network enables the assessment participants 
unrestricted access to the application’s user interface 
function. 


3. Does the proposed unit self-assessment package 
provide project managers with the information and 
tools necessary to track and improve program health 
and customer satisfaction? 


The recommended self-assessment package, consisting of 
the CCAWS/AGMS Unit Self-assessment and the prototype software 
application provides the project office’s management personnel 
with sufficient information and adequate tools to evaluate, 
monitor, and improve the program health and internal customer 
satisfaction of the project office. The analysis of feedback 
collected during the candidate assessment evaluation and the 


prototype software testing concerning the overall assessment 


package addresses the following items: 





a. Data Analysis 

The types and depth of the analyses incorporated by 
the unit self-assessment package are sufficient to meet the 
sponsors’ requirements to generate statistically relevant data 
that provide meaningful feedback concerning the status of the 
program health and internal customer satisfaction of the 
project office. 

b. Data Presentation 

The unit self-assessment package incorporates the 
use of computer generated charts and graphs to present the 
results of generated feedback and analyses in a manner that is 
clear, concise, and useful to the project office’s management 
team. These charts and graphs can be either viewed on screen 
from any available workstation connected to the organization’s 
LAN or printed to support additional analyses, discussion, or 
other purposed as desired by management. 

c. Data Storage 

The unit self-assessment package exceeds the 
sponsors’ requirement to store collected data for a period of 
not less than five years. The assessment application is 
capable of storing data collected by the assessment process 
for an unlimited period. If, however, data are accumulated 
for a period greater than the five year requirement, this 
retention of excess data may cause the presentation of 
analyses to become confused and cluttered and may also result 
in a noticeable decrease in the speed at which these charts 


and graphs are displayed. 


C. RECOMMENDATIONS 


Efficiency in the acquisition of major defense systems is 
of paramount importance, given today’s environment of 


declining budgets and Congressional and public skepticism. 
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To achieve this efficiency, the project office, as the 
responsible agency for such acquisitions, must in turn perform 
its designated functions both effectively and efficiently. 
This can only be accomplished by establishing systems within 
the program structure to assess, monitor, and improve the 
critical factors that determine the efficiency and the 
effective operation of the project office. The two most 
important indicators reflecting the efficiency and 
effectiveness of the project office are program health and 
customer satisfaction. To this end, this study recommends the 
following items: 


1. Conduct Periodic Assessments Using the CCAWS/AGMS 
Unit Self-assessment Package 


The CCAWS/AGMS Unit Self-assessment package, consisting 
of the CCAWS/AGMS Unit Self-assessment and the prototype 
software application, is both an effective and efficient 
method to measure the program health and internal customer 
satisfaction of a project management office. Assessments 
should be conducted semi-annually using this automated 
assessment instrument to identify and monitor areas needing 
improvement with respect to the critical factors that 
determine the program health and internal customer 
satisfaction of the project office. 


2. Conduct Periodic Reviews of the CCAWS/AGMS Unit Self- 
assessment Functional Areas and Items 


From time to time occurrences may arise in which the 
strategy, goals, objectives, priorities, or functions of the 
project office may change due to external influences or as the 
result of internal initiatives. To ensure the accurate 
measurement of the critical factors that determine the program 
health and internal customer satisfaction of the project 
office, the assessment instrument used to perform this 


measurement should be modified to reflect these changes, as 
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necessary. To this end, the CCAWS/AGMS Unit Self-assessment 
should be reviewed annually to ensure critical changes in the 
functioning of the project office are reflected in a timely 


Manner. 


3. Institute a Separate Assessment Focusing on Customer 
Satisfaction 


As stated earlier, one of the two most important 
indicators of the efficient and effective functioning of the 
project management office is customer satisfaction. The 
CCAWS/AGMS Unit Self-assessment, however, measures only the 
customer satisfaction internal to the project office and does 
not address the level of satisfaction experienced by the 
project office’s external customers. Given the importance of 
this factor, an additional assessment instrument should be 
designed for the purpose of measuring this factor. This 
assessment, once developed, should be conducted using the 


existing prototype software application recommended by this 
study. 


D. CONTRIBUTIONS 


This thesis provides the project manager with a viable 
model depicting the critical organizational design factors 
impacting on the program health and internal customer 
satisfaction of the military project office. This study also 
provides the project manager an implementing instrument, based 
on the above model, with which to assess the level of program 
health and internal customer satisfaction of the project 
office: Finally, this research effort has produced an 
independent software application, specifically designed to 
automate the self-assessment process within a military project 
management office which will significantly reduce the man- 


hours associated with the self-assessment process. 
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E. RESEARCH LIMITATIONS 

This section describes the limitations concerning this 
research effort surfaced during the candidate assessment 
evaluation and the prototype software application testing. 
The items listed here are in addition to those previously 
discussed in Chapter I of this thesis. 

1. Reliability and Validity of Assessment Items 

The CCAWS/AGMS Unit Self-assessment is a= pilot 
instrument. Many of the items used in the assessment are 
extractions or modifications from the USAREUR Organizational 
Diagnostic Survey (modified). These items have not been 
tested to determine their reliability or validity to measure 
the factors with which they have been grouped. They are, 
however, deemed to have face validity, based on responses from 
various personnel with longevity in the project management 
discipline. Additionally, items which are included in factor 
areas that were added based on feedback acquired during the 
candidate assessment evaluation have not been evaluated and 
therefore cannot be said to contain face validity. This 
limitation is applicable to those items within the customer 
satisfaction, integrated product team, and products/services 
functional areas. 

2. Software Compatibility 

The prototype software application is designed to operate 
effectively across the organization’s LAN. The documentation 
supporting the development of the prototype application has 
confirmed its compatibility with the types and brands of LANs 
used in both the CCAWS and AGMS project offices. A physical 
test, however, to verify the application’s compatibility with 
the LANs was not conducted due to time constraints and the 
candidate assessment evaluation and software testing methods 


chosen by this researcher. 


F. RECOMMENDATIONS FOR FURTHER STUDY 


1. CCAWS/AGMS Unit Self-assessment Item Reliability and 
Validity Testing 


The assessment instrument recommended by this study, the 
CCAWS/AGMS Unit Self-assessment, contains face validity only, 
based on the expert opinions of personnel with longevity in 
the field of project management, and has not been subjected to 
any form of psychometric testing or evaluation. A study of 
the reliability and validity of the items incorporated in the 
CCAWS/AGMS Unit Self-assessment will confirm or deny the 
effectiveness of these items to accurately measure the program 
health and customer satisfaction of a project management 
office. 

2. Customer Satisfaction Survey Development 

The CCAWS/AGMS Unit Self-assessment provides a means with 
which to measure the program health and internal customer 
satisfaction of a project management office. This instrument 
does not, however, address the sponsors’ requirement to 
measure the external customers’ level of satisfaction with the 
products and services delivered by the project office. 
Research in this area would result in the development of an 
assessment instrument designed for use by the project 
management office to measure the satisfaction of its external 


customers. 
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APPENDIX A: THE USAREUR ORGANIZATIONAL DIAGNOSTIC SURVEY 


Group: 





(MODIFIED) 


1. Strategy (Mission - Goals - Objectives) 


.1 - To what extent do you understand the mission of 


the Project Office? (Scale 1) 


-2 - To what extent do you understand the objectives 


of the Project Office? (Scale 1) 


.3 ~ To what extent are the objectives of the Project 


Office realistic? (Scale 1) 





-4 - To what extent do the objectives of the Project 


Office support the goals of Program Executive 
Office? (Scale 1) 


.5 - To what extent are the people in the Project 


Office committed to achieving its objectives? 
(Scale 1) 


-6 - I am satisfied with my role in setting 


organizational objectives. (Scale 2) 


.7 - To what extent are the Project Office's 


objectives for mission accomplishment consistent 
with the demands of its environment? (Scale 1) 


8 - To what extent are the Project Office's 
objectives for mission accomplishment consistent 
with the availability of its resources? 

(Scale 1) 
9 - To what extent does the work actually performed 


support the Project Office's stated objectives? 
(Scale 1) 


.10 - The strategy of the Project Office could be 


improved by: (open) 


Group: 2. Task 


2s 


1 - I know what tasks I am responsible for 
accomplishing. (Scale 2) 
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To what extent do your assigned tasks include 
work that you consider to be outside your area of 
responsibility? (Scale 1) 


To what extent are your assigned task deadlines 
reasonable? (Scale 1) 


To what extent does the accomplishment of your 
duties depend on the coordinated input of other 
people? (Scale 1) 


To what extent are you able to plan your 
activities for the following week? (Scale 1) 


To what extent do your activities occur as you 
have them planned? (Scale 1) 


I have sufficient authority to accomplish my 
assigned tasks. (Scale 2) 


To what extent do the Project Office's procedures 
hinder task accomplishment? (Scale 1) 


Task accomplishment in the Project Office could 
be improved by: (open) 


Structure 


To what extent does the Project Office's present 
structure (the way people are divided up to do 
work) facilitate work? (Scale 1) 


To what extent does the Project Office's present 
structure facilitate the flow of information? 
(Scale 1) 


To what extent does the Project Office's present 
structure facilitate decision making? 
(Scale 1) 


Our informal work structure complements the 
established formal structure. (Scale 2) 


To what extent are individual responsibilities 
defined in the Project Office? (Scale 1) 


To what extent is the accountability for final 
products clear? (Scale 1) 
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3.7 - To what extent are people who are responsible for 
task accomplishment provided the necessary 
resources? (Scale 1) 


3.8 - To what extent do people who are responsible for 
task accomplishment have the necessary authority? 
(Scale 1) 


3.9 - To what extent are there barriers to the lateral 
flow of information in the Project Office? 
(Scale 1) 


3.10 - To what extent is necessary information passed 
up and down the chain of command? (Scale 1) 


3.11 - To what extent do the policies of the Project 
Office hinder mission accomplishment? (Scale 1) 


3.12 - The way we are divided up to do things and the 
procedures we follow could be improved by: 
(open) 


Group: 4. People 


aa 
ray 
| 


The people in the Project Office have the skills 
and knowledge necessary to accomplish their 
assigned tasks. (Scale 2) 


4.2 - The people in the Project Office clearly express 
their knowledge of and support for the Project 
office's objectives. (Scale 2) 


4.3 - Personnel in the Project Office understand their 
jobs and how they contribute to the overall 
organizational effort. (Scale 2) 


4.4 - There is a high degree of personal commitment to 
the Project Office. (Scale 2) 


4.5 - The people in the Project Office are warm and 
friendly toward each other. (Scale 2) 


4.6 - The Project Office is seen as a good place to 


work because its people are treated right. 
(Scale 2) 
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To what extent does the structure of the Project 
Office help people as they do their work. 
(Scale 1) 


The following things need to be done in order to 
improve the fit between the kind of people we now 
have and the kind we ought to have: (open) 


Rewards 


To what extent are rewards in the Project Office 
contingent upon results rather than tenure or 
rank? (Scale 1) 


The criteria for receiving rewards are clear to 
me. (Scale 2) 


To what extent do you understand the performance 
standards of the Project Office? (Scale 1) 


To what extent do supervisors in the Project 
Office inform their subordinates of their duty 
performance? (Scale 1) 


To what extent are you satisfied with the award 
system in the Project Office. (Scale 1) 


To what extent does the reward system in the 
Project Office motivate people to achieve better 
results. (Scale 1) 


The reward system in the Project Office could be 
improved by: (open) 


Processes 

Communication is good throughout the Project 
Office. (i.e. the people know what is going on). 
(Scale 2) 


Coordination is good throughout the Project 
Office. (Scale 2) 


To what extent is conflict dealt with openly and 
managed constructively in the Project Office? 
(Scale 1) 
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Group: 


To what extent are clear individual objectives 
and performance criteria established for each 
person in the Project Office? (Scale 1) 


Decisions are made at the most appropriate level 
in the Project Office. (Scale 2) 


To what extent do the formally established 
policies and procedures in the Project Office 
match what people actually do? (Scale 1) 


To what extent do people in the Project Office 
understand which decisions they can make 
themselves and which ones must involve other 
people? (Scale 1) 


The informal power structure of the Project 
Office complements the formal structure. 
(Scale 2) 


The processes of the Project Office 
(communication, decision making, conflict 
management, individual role clarification) could 
be improved by: (open) 


Leadership 


To what extent does the leadership of the Project 
Office provide clear guidance on tasks that must 
be accomplished? (Scale 1) 


To what extent is the leadership of the Project 
Office proactive (focused on the future) rather 
than reactive? (Scale 1) 


The leadership of the Project Office demonstrates 
that they trust me to do my job well. (Scale 2) ‘ 


The informal leaders in the Project Office 
complement the formal leadership. (Scale 2) 


To what extent is the leadership of the Project 
Office respected by the people in the Project 
Office? (Scale 1) 


To what extent are the leaders of the Project 
Office aware of what is going on "in the 
trenches" of the Project Office? (Scale 1) 
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Group 8. 


Ss 


Group: 


1 


The leaders in the Project Office are genuinely 
people oriented. (Scale 2) 


The leadership of the Project Office could be 
improved by: (open) 


Technology 


To what extent is the equipment of the Project 
Office (computers, word processors, copiers, 
etc.) sufficient to accomplish its mission? 
(Scale 1) 


To what extent does the Project Office's budget 
reflect the funds necessary to purchase needed 
equipment? (Scale 1) 


The Project Office makes full use of the 
equipment it does have. (Scale 2) 


To what extent are personnel in the Project 
Office trained in the proper use and maintenance 
of their equipment? (Scale 1) 


To what extent is routine, repetitive work in the 
Project Office automated? (Scale 1) 


Our management information system (MIS) supports 
the Project Office rather than vice versa. 
(Scale 2) 


To what extent is additional equipment and timely 
external technical support available? (Scale 1) 


The technical support of the Project Office can 
be improved by: (open) 


General 


The things I like most about the Project Office 
are: (open) 


The things I like least about the Project Office 
are: (open) 


If I were in charge of the Project Office, I 
would: (open) 
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Response Scales 





Scale 1 Value Scale 2 












Totally 


Substantially 


Strongly Agree 
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Strongly Disagree 
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APPENDIX B: THE USAREUR ORGANIZATIONAL DIAGNOSTIC SURVEY 
Group: 1. Strategy (Mission - Goals - Objectives) 


1.1 - I understand the strategy (mission, goals, 
objectives)of this organization. 


1.2 - The organization has a set of realistic 
objectives which support the goals of the next 
higher organization. 


1.3 - This organization's strategy will project the 
organization into the future in the most 
effective manner. 


1.4 - There is a high degree of commitment to this 
organization's mission and objectives. 


1.5 - I am satisfied with my role in setting 
organizational objectives. 


1.6 - This organization's strategy for mission 
accomplishment (link between mission, goals, 
objectives) is consistent with the demands of its 
environment and resources. 


1.7 - The work actually done in this organization 
matches that dictated by its strategy. 


1.8 - The strategy of this organization could be 
improved by: 


Group: 2. Task 


2.1 - I know what tasks I am responsible for 
accomplishing. 


2.2 - Most of the tasks I am assigned fall within what 
I consider my area of responsibility. 


2.3 - The suspense dates associated with most of my 
assigned tasks are reasonable. 


2.4 - My task accomplishment is usually dependent upon 
input from other people. 


2.5 - I can usually plan ahead what my tasks for the 
following day will be. 
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Group: 3. 


Group: 4. 


I have enough authority to accomplish my assigned 
tasks. 


Our procedures and management practices 
facilitate task accomplishment. 


Task accomplishment in this organization could be 
improved by: 


Structure 
Our present structure (the way we are divided up 
to do work -- the wiring diagram) facilitates 


work, information flow, and decision making. 


Our formal work structure is the same as our 
informal structure. 


There is a clearly understood definition of who 
does what within this organization. 


Accountability for final products is clear. 
People who are responsible for task 
accomplishment have the necessary resources and 


authority. 


Information flows easily across boundaries in 
this organization. 


Our rules and policies help us get the job done. 
The way we are divided up to do things and the 
procedures we follow to do them could be improved 
by: 

People 

The personnel in this organization have the 
skills and knowledge necessary to accomplish 


their assigned tasks. 


The people in this organization clearly express 
their knowledge of and support for the 
organizational strategy. 
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Personnel in this organization understand their 
jobs and how they contribute to the overall 
organizational effort. 


There is a high degree of personal commitment to 
this organization. 


Organizational members are warm and friendly 
toward each other. 


This organization is seen as a good place to work 
because it treats its people right. 


The structure/process arrangements in this 
organization help people as they do their work. 


The following things need to be done in order to 
improve the fit between the kind of people we now 
have and the kind we ought to have: 


Rewards 


Rewards in this organization are contingent upon 
results rather than tenure or rank. 


The criteria for receiving rewards are clear to 
me. 


Performance standards for individuals in this 
organization are clear to all concerned. 


It is easy to find out how you are doing in this 
organization. 


I am satisfied with the award system in this 
organization. 


The reward system supports the people in their 
desires. 


The present reward system motivates people to 
achieve better results. 


The reward system in this organization could be 
improved by: 





Group: 


Group: 


Processes 


This organization has good lateral communication 
(i.e. the people know what is going on). 


Conflict is dealt with openly and is managed 
constructively in this organization. 


Clear individual objectives and performance 
criteria are established for each position in 
this organization. 


Decisions are made at the most appropriate level 
in this organization. 


Our formally stated methods and practices match 
what people actually do. 


People in this organization understand which 
decisions they can make and which ones must 
involve others. 


The formal power structure is the same as the 
informal power structure. 


The processes of communication, decision making, 
conflict management, and individual role 
clarification could be improved by: 


Leadership 

The leadership of this organization provides us 
with sufficient clear guidance on tasks we must 
accomplish. 

The leadership of this organization focuses on 
the future (proactive) rather than the past 


(reactive). 


The leadership of this organization demonstrates 
that it trusts me to do my job well. 


The "formal" leaders of this organization are 
also the "informal". 


The leadership of this organization is respected 
by the organization. 
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The leadership of this organization is aware of 
what is going on "in the trenches". 


The leaders in this organization are genuinely 
people oriented. 


The leadership of this organization could be 
improved by: 


Technology 


This organization has sufficient equipment 
(computers, word processors, calculators, etc.) 
To accomplish its mission. 


The money to purchase new equipment that we do 
need but do not have is reflected in our budget. 


This organization makes full use of the equipment 
it does have. 


The personnel in this office have sufficient 


knowledge to operate and maintain the equipment 
it does have. 


Most of our routine, repetitive work is 
automated. 


Our management information system (MIS) supports 
our organization rather than vice versa. 


If this organization does not have the 
technological support needed, it is available 
from another organization on a timely basis. 


The technological support and equipment we need 
to do a better job fall into these areas: 


General 
The thing I like most about this organization is: 


The thing I like least about this organization 
is: 


If I were in charge of this organization, I 
would: 
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APPENDIX C: THE CCAWS/AGMS UNIT SELF-ASSESSMENT 
Mission/Objectives 


To what extent do you understand the mission of 
the Project Office? (Scale 1) 


To what extent do you understand the objectives 
of your office? (Scale 1) 


I agree with/support the objectives of my office? 
(Scale 2) 


To what extent does the work actually performed 
support your office's stated objectives? 
(Scale 1) 


Task Requirements 


I know what tasks I am responsible for 
accomplishing. (Scale 2) 


To what extent are your assigned task deadlines 
reasonable? (Scale 1) 


To what extent does the accomplishment of your 
duties depend on input from other people? 
(Scale 1) 


To what extent are you able to plan your 
activities for the following week? (Scale 1) 
To what extent do your activities occur as you 
have them planned? (Scale 1) 


I have sufficient authority to accomplish my 
assigned tasks. (Scale 2) 


an 


To what extent do the procedures in your office 
hinder productivity? (Scale 1) 





2.8 - Task accomplishment in my office could be 
improved by: (open) 





Group: 


Leo) 


[e) 


1a) 


Group: 


3. Structure 


-1 - To what extent does the your office's present 


structure (the way people are divided up to do 
work) facilitate task accomplishment? (Scale 1) 


-2 - To what extent does the Project Office's present 


structure facilitate the flow of information? 
(Scale 1) 


-3 - The informal work structure in our office 


complements the established formal structure. 
(Scale 2) 


od 
| 


To what extent are your individual 
responsibilities defined? (Scale 1) 


-5 - To what extent is the accountability for final 


products clear? (Scale 1) 


-6 - To what extent are you provided the necessary 


resources to accomplish the tasks for which you 
are responsible? (Scale 1) 


.7 - I have the necessary authority to accomplish the 


tasks for which I am responsible? (Scale 2) 


-8 - To what extent are there barriers to the lateral 


flow of information in the Project Office? 
(Scale 1) 


-9 - To what extent do you receive the information 


necessary to perform your job? (Scale 1) 


.10 - To what extent do the policies in your office 


hinder mission accomplishment? (Scale 1) 


-l1 - The way we are divided up to do things and the 


procedures we follow could be improved by: 
(open) 


4. People 


-l - I have been adequately trained in the skilis 


and knowledge necessary to accomplish my 
assigned tasks. (Scale 2) 


The people in my office clearly express 
their knowledge of and support for the office's 
objectives. (Scale 2) 


I understand how my work contributes to the 
overall organizational effort. (Scale 2) 


The people in our office are warm and 
friendly toward each other. (Scale 2) 


Our office is seen as a good place to 
work because its people are treated well. 
(Scale 2) 


To what extent does the structure of your 
office help people as they do their work. 
(Scale 1) 


The following things need to be done in order to 
improve the fit between the kind of people we now 
have and the kind we ought to have: (open) 


Integrated Product Teams (IPTs) 


I am a member of a product oriented IPT. 
(Scale 2) 


Conflicts are openly resolved between members of 
the IPT. (Scale 2) 


To what extent do you understand the objectives 
of your IPT? (Scale 1) 


To what extent does your IPT include members 
representing all of the functional areas critical 
to the product’s success? (Scale 1) 


To what extent do you feel the members of 
the IPT are focused on common objectives? 
(Scale 1) 


To what extent are members of the IPT held 
accountable for areas in which they are 
responsible? (Scale 1) 


I understand how my participation in the IPT 
relates to the accomplishment of its objectives. 
(Scale 2) 





Group: 


fart 


ao abieks 


The effectiveness of the IPT could be improved 
by: (open) 


Processes 


Communication is good in our office. (i.e. the 
people know what is going on). (Scale 2) 


Coordination is good between offices within the 
Project Office. (Scale 2) 


To what extent are personal conflicts dealt with 
openly in your office? 
(Scale 1) 


To what extent are clear individual objectives 
established for each person in your office? 
(Scale 1) 


To what extent are personal conflicts managed 
constructively in your office? 
(Scale 1) 


To what extent are performance criteria 
established for each person in your office? 
(Scale 1) 


Decisions are made at the most appropriate level 
in my office. (Scale 2) 


To what extent do the formally established 
policies and procedures in your office match what 
people actually do? (Scale 1) 


To what extent do you understand which decisions 
you can make yourself and which ones must involve 
other people? (Scale 1) 


-10 - The informal power structure within our office 


complements the formal structure. (Scale 2) 


- The processes use in our office (communication, 


decision making, conflict management, individual 
role clarification) could be improved by: (open) 
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Leadership 


To what extent does your immediate supervisor 
provide clear guidance on tasks that must 
be accomplished? (Scale 1) 


To what extent are the leaders in your office 
proactive (focused on the future) rather than 
reactive? (Scale 1) 


My immediate supervisor demonstrates that he/she 
trusts me to do my job well. (Scale 2) 


The informal leaders in my office complement the 
formal leadership. (Scale 2) 


To what extent do you respect the leaders in your 
office? (Scale 1) 


To what extent is your immediate supervisor aware 
of what is going on "in the trenches" of your 
Office? (Scale 1) 


My immediate supervisor is genuinely people 
oriented. (Scale 2) 


The leadership in our office could be more 
effective by: (open) 


Technology 


To what extent is the equipment in your office 
(computers, word processors, copiers, etc.) 
sufficient to accomplish its mission? 

(Scale 1) 


Our office makes full use of the equipment it 
does have. (Scale 2) 


To what extent does the basic approach to task 
accomplishment fit the required tasks and the 
resources available? (Scale 1) 


To what extent have you been trained in the 
proper use and maintenance of the equipment 
you need to perform your job? (Scale 1) 





8.5 - To what extent is routine, repetitive work in 
your office automated? (Scale 1) 


8.6 - The Management Information System (MIS) in the 
Project Office supports our office rather than 
vice versa. (Scale 2) 


8.7 - The technology of the Project Office can be 
improved by: (open) 


Group 9. Products/Services 


9.1 - To what extent are the products/services 
delivered by your office designed to meet 
customer expectations? (Scale 1) 


9.2 - Our office delivers professional products/ 
services to its customers. (Scale 2) 


9.3 - To what extent is your office product/service 
oriented? (Scale 1) 


9.4 - To what extent do the products/services delivered 
by your office require rework? (Scale 1) 


9.5 - To what extent are there procedures in your 
office to ensure the output of quality products 
and services? (Scale 1) 


9.6 - The products and services delivered by our 
office could be improved by: (open) 


Group 10. Customer Satisfaction 


10.1 - To what extent are you satisfied with the 
products you receive from other offices within 
the PMO? (Scale 1) 


10.2 - To what extent are you satisfied with the 
services performed by other offices within the 
PMO? (Scale 1) 


10.3 - To what extent do the products/services you 
receive from other offices within the PMO 
require rework? (Scale 1) 
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10.4 - To what extent does the accomplishment of your 
assigned tasks depend on the products/services 
you receive from other offices within the PMO? 
(Scale 1) 


10.5 - In your opinion, which office(s) within the PMO 
provides quality products/services to its 
internal customers? (Open) 


10.6 - In your opinion, which office(s) within the PMO 
provides poor products/services to its internal 
customers? (Open) 


10.7 - My satisfaction with the products/services 
received from other offices within the PMO could 
be improved by: (open) 

Group: 11. General 


11.1 - The things I like most about working in my 
office are: (open) 


11.2 - The things I like least about working in my 
office are: (open) 


11.3 - If I were in charge of my office, section, 
branch, or division, I would: (open) 


Response Scales 


ere 
[Sibeteatiaity [=| 
[ Reasonamly | =a 
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APPENDIX D: UNIT SELF-ASSESSMENT ITEM SURVEY 


Respondent #: __ Name: 
Organization: Office: 
Position: 


For each Item: 


Is the wording: 
O O O O O 


J_ 


Ambiguous Clear 
O | O O Ey 


tt 


Wordy Concise 


Is the content: 


[1 ry) I ee) O 
J_ 
Confusing Understandable 
O O O O im 
$$] 
Irrelevant Applicable 
O O O O O 


a at 


Offensive Suitable 


Is the Scale: 


O O CJ O O 


| 


Improper Appropriate 
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APPENDIX E: UNIT ASSESSMENT AND SOFTWARE SURVEY 


Respondent #: Name: 
Organization: Office: 
Position: 


A. Unit Assessment: 


1. The instructions provided on how to complete the 
assessment are —__ 


O | EI O O 
ee SSS 
Confusing Clear 
O O O O O 


-W———__4+—-I__+——\—_+] 


Incomplete Complete 


2. The assessment itself is 


O O O O O 
a ee ee 
Too Short About Right Too Long 
O EJ CO C] O 
a 
Incomplete Comprehensive 
3. _—CO'déE:«ésPUCKE:«SCASSESSment questions are 
for a person at my level in the Project Office. 
O oO EI O O 
aa 
None Few Some Most All 
OC 0 O oO O 


Inappropriate Appropriate 





4. What questions/areas do you feel are important 
indicators of program health that are not included in the 
assessment? 


5. How long did it take you to complete the 
assessment? 


minutes. 


6. Please, provide any General Comments you may have 
about the assessment. 


B. Software: 


1. The instructions provided on how to use the 
software are: 


EJ O C oO O] 

——_—_$ S$ 
Confusing Clear 

La CO O i) C) 

pa a ee 
Incomplete Complete 
2. Is the Microsoft Windows environment installed on 


your computer workstation? 


Yes / No 
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3. The attributes used in the assessment software 
(windows, push buttons, drag and drop, etc..) were 
to me. 


O C 0 LC] C] 
t—_+-—— tH 


Unfamiliar Familiar 


4. Compared to a "paper and pencil" assessment, the 
computerized assessment is: 


CO C] CL LC C 
($$$ 


Less time consuming More time consuming 


O O O L) C 
-———_1t>—_————_-1—— 


Easier to complete Harder to complete 


5. Did you encounter any difficulties using the 
software? 


Yes / No 


If yes, please describe the difficulties. 


6. What, if any, improvements to the software would 
you suggest? 


7. Please, provide any General Comments you may have 
about the software. 
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APPENDIX F: UNIT SELF-ASSESSMENT ITEM SURVEY DATA 


The data collected from Survey #2 concerning the effectiveness 
of the candidate self-assessment items to measure the program 
health and customer satisfaction of a project management office are 


summarized in this appendix. The items are listed as follows: 





Table Functional Area Items 








Table 7 Strategy 1.1-1.10 
Table 8 Task 2.1-2.9 
Table 9 Structure 3.1-3.12 
Table 10 People 41-48 
Table 11 Rewards 5.1-5.7 
Table 12 Processes 6.1 - 6.9. 
Table 13 Leadership 7.1-7.8 
Table 14 Technology 8.1-8.8 


Table 15 General 9.1-9.3 


Table 6: Directory of Survey #2 Data Tables. 
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(Strategy) 


Assessment Item Survey Data - Group 1 


Table 7 
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Assessment Item Survey Data - Group 2 


Table 8 
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(Structure) 


Assessment Item Survey Data - Group 3 


Table 9 


130 











%8T 66 








%*001 


4 





001 





%001 


2 
ba 
a 











%00T 


%001 

















% 


























%001 


%Iv'66 


007 007 007 00% 



























(People) 


Assessment Item Survey Data - Group 4 


Table 10 
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Assessment Item Survey Data - Group 5 (Rewards) 


Table il 


132 





(Processes) 











YrS'66 





001 














%SS'86 





Y8E66 


%00I HOO %O0I ie a %459'S6 
%O0T 


001 








133 











YoO0OE "001 %SI'S6 
Ea Ee ee 


00% 00% 00% 00% 00°% Té6'T 















Assessment Item Survey Data - Group 6 
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Assessment Item Survey Data - Group 7 


Table 13 


134 









%001 8766 


%00T 001 %001 %001 %001 % 001 
000 


%W8E 66 


%E9'66 


001 


%001 } voor | %O0t 26001 %O0I %00I 
a 00% 00% 00% 00% 
Lun 


























*@001 %00T %0OT 
















%001 
































00% 











(Technology) 


Assessment Item Survey Data - Group 8 


Table 14 
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APPENDIX G: QUALITATIVE RESPONSE SUMMARY: UNIT SELF- 
ASSESSMENT ITEM SURVEY 


Does not apply to secretary position. 


Too general. Reduce scope to appropriate 
within the organization. 


Too general. Reduce scope to appropriate 
within the organization. 


Change the word "objectives" to "vision". 


Too general. Reduce scope to appropriate level 
within the organization. 


Change the word "objectives" to "vision". (2) 


Irrelevant because these are fixed at higher 
levels. Possible to ask about sub-objectives that 
support these objectives. (2) 


Too general. Reduce scope to appropriate level 
within the organization. 


Irrelevant question, delete. 


A majority of the people will not know the goals of 
the PEO. (3) 


Too general. Reduce scope to appropriate 
within the organization. 


Change the word "objectives" to "vision". 


Irrelevant question, delete. 


Not applicable to all job positions. (4) 


Change the words "objectives for mission 
accomplishment" to "vision". 





Item 


Item 


Item 


Item 


Item 


210s 


Environment and Demands are not clear. Question 


may need supporting questions preceding it to make 
it clear. (9) 


What “strategy”, be specific. 


Reword: s<sthe PMO environment suitable tc 
accomplish mission objectives... 


Irrelevant question, delete. 


Reword: Does the PMO have the resources necessary 
to accomplish its mission? 


Define "Availability of its resources". People, 
money, etc... (3) 


Change the word "objectives" to "vision". 


Remove the word “Stated” from the question. 
Possibly use goals/objectives. 


Change the word "strategy" to "strategy/vision". 
Define "Strategy". (5) 


Reword: How could the PMO better accomplish its 
mission? 


Look at asking what the PMO does well also. 


Scale: Reasonable/Moderate not clear for this 
question. Confuses the user. Look for better 
words. Suggest “little”. 


Delete question (could be a grievance). 


Reword: as a statement. "Tasks are assigned". Use 
scale #2. 


Delete the word "Task". 


Wording. Approach from the viewpoint of 
unreasonable. 


Reword: Are your task deadlines reasonable most of 
the time (or) how much of the time? There is a 
certain amount of crisis management inherent in 
this business. 
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Item 


Item 


Item 


Item 


Item 


Item 


Item 


Item 


Not specific. Find a way to eliminate crisis 
generated tasks. Maybe use the word normal or 
routine. (3) 


Reword as a statement. "Tasks are assigned..." 
Use scale #2. 


Change the word duties to tasks (consistent with 
group). 


Coordinated input is confusing. 


Reword: "Do you plan your activities..." 


“By who” delegated. 


Solicit both problems and solutions, also things 
working well. (2) 


Scale down to branch office. Cannot be answered 
accurately by most people in the organization as 
written. 


Too general. Reduce scope to appropriate level 
within the organization. 


Change the word "work" to "your assigned tasks". 


Type of flow is confusing (lateral or vertical). 


Who/What level of decision making does this 
question refer to? (2) 


Define "informal work structure". (2) 


Is this question referring to the Integrated 
Product Teams or the informal structure? 


Change structure to environment. Change the word 
"Formal" to "established PMO structure”. 
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Item 


Item 


Item 


Item 


Item 


Item 


Item 


Item 


Item 


Item 


.10: 


~Lis 


.12: 


Too general. Reduce scope to appropriate level 


within the organization. 


Define “final products”. Product 


refers to the 


final deliverable by the PMO. Replace the word 


"product". (2) 


Personalize: Are you held accountable? 


Needs to be more direct. 


Too general. Reduce scope to appropriate level 


within the organization. 


Too general. Reduce scope to appropriate levei 


within the organization. 


Assumes all people know the decision process. 


Define lateral (example- across functional areas). 


(3) 


Add "mission essential" or "important" information. 


Define "Chain of Command". 


Too general. Reduce scope to appropriate level 


within the organization. 


Not specific (how - at what ievel). 


Look at asking what the PMO does well also. 


Clearly "understand" and "support". 
two separate questions. 


The word structure is confusing. 
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This should be 


Item 


Item 


Item 


Item 


Item 


Item 


Item 


Item 


Item 


Item 


Look at asking what the PMO does well also. 


Reword: drop the ending. Focus of the question 
should be on equity. Is the awards system 
equitable? The use of rank or tenure restricts the 
question. What about sex, race, etc... Use the 


word "performance" instead of "results". 


Address individual performance standards instead of 
overall office standards. (3) 


Suggest changing this question to read "How many 
times in the past 12 months has your immediate 
supervisor counseled you on your duty 
performance?". (2) 


Should refer to the office, not the PMO. (4) 
Use the word "job" instead of "duty". 


Add "Given the current constraints on the awards 
process",... 


Personalize to individual job. (Personal and 
professional environment motivate you to achieve.) 


Look at asking what the PMO does well also. 


Specify "official" communications. 


“Of What” (across functional areas?) (Does it 
achieve results?) Possibly two separate questions 
needed. 


Define type of conflict (Mission, Role, Personal?) 
(3) 
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Item 


Item 


Item 


Item 


Item 


Item 


Item 


Item 


Item 


Item 


Delete. Irrelevant because each individual 
establishes his/her own objectives, based on an 
agreement between the individual and his/her 
immediate supervisor. 


Personalize: ask about the respondent. (3) 


Too general. Reduce scope to appropriate level 
within the organization. 


Assumes an informal power structure exists. 


Look at asking what the PMO does well also. 


Too general, needs to focus on the _ office 
leadership, first line supervisor, not the PMO. (3) 


Define "leadership" or reduce scope. (2) 


Too general. Reduce scope to appropriate level 
within the organization. 


Too general. Reduce scope to appropriate level 
within the organization. 


Do the IPTs (Integrated Product Teams) complement 
the chain of command? 


Assumes informal leadership exists. 


Too general. Reduce scope to appropriate level 
within the organization. 


Too general. Reduce scope to appropriate level 
within the organization. 
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Item 


Item 


Item 


Item 


Item 


Item 


Item 


Item 


Item 


Item 


Item 


Too general. Reduce scope to appropriate level 
within the organization. 


Too general. Reduce scope to appropriate level 
within the organization. 


Look at asking what the PMO does well also. 


Irrelevant question, delete. 


Irrelevant question, delete. 


Irrelevant question, delete. 


Define the type of equipment “Is it talking about 
the product or the admin support equipment?” 


Possibly two questions being asked. 


Irrelevant question, delete. 


Add a question that focuses on the "Paperless 
environment”. 


Define "MIS". (2) 


Define external support or delete. (2) 


Irrelevant question, delete. 


Define technical support. Is this the equipment 
itself or the tech spt group within the PMO? 


Look at asking what the PMO does well also. 


Look at asking what the PMO does well also. 


Look at asking what the PMO does well also. 
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Item 9.3: 


Look at asking what the PMO does well also. 


Additions: 


Item in group 1: Does the Project Office's vision 
support our customers. 


Item in group 2: What causes you the most stress in 
your daily job? 


Item in group 3: Is there a proper balance between 
functional and product line roles 
responsibilities) ? 


Item in group 4: What makes the Integrated Product 
Teams (IPT) work? What needs to be 
improved? 





Item in group 7: Is the leadership/organization 
responsive to the needs of the customer? 


Item in group 8: Do you feel the Project office 
produces quality products, provides quality 
services? 


Item in group 8: Do you feel the Project office 
produces quality end items (HW/SW) ? 


Item in group 8: How effectively do you feel the 
Project office communicates, works with, 
contractors? Other customers? 


Item: Focused on the quality of the counseling. 
Item: Do you know the goals of the PEO? 


Item: How satisfied are you with the support you 
receive from the other offices within the PMO? 


Item: To what extent is the awards system limited 
by the command (external to the PMO). 
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APPENDIX H: QUALITATIVE RESPONSE SUMMARY: UNIT SELF- 


ASSESSMENT AND SOFTWARE SURVEY 


Unit Self-assessment: 


Item A4: 


What questions/areas do you feel are important 


indicators of program health that are not included in the 


assessment? 


° 
° 

Item A6: 

about the 


Relationships with matrix/external organizations 
(possibly under structure). 


Questions that focus on the project office’s 
relationship with the PEO & MICOM. 


As noted during sample/walk through of the 
assessment-worker perceptions/knowledge of mission, 
goals, vision, and their role/position in achieving 
them. How is their worth defined, validated, and 
preserved. Do they make a difference? Are they 
part of the team , or not? 


Training - computer literate. 


Customer satisfaction related questions on internal 
customers. 


Participation in "off-site" activities. The turn 
out we have shows that a lot of employees don't 
want to participate. 


Leadership and technical. 


Add assessment of individual training. 


Please, provide any general comments you may have 


assessment instrument. 


The assessment was, with the exception of one 
question, clear and concise. Some of the questions 
were inappropriate for a support contractor like 
myself, but, in general, the assessment will be a 
useful tool for Project Management in assessing the 
level of success of their leadership. 


Much shorter and to the point than the previous 
survey. 


The assessment was quick, easy to understand, and 
appropriate to this project office. 


Very positive. I think the tool is solid. 





Asks the right kind of questions and not too many. 
Is not tiring and aggravating to take. 


Easy to use, painless. 


Minor modifications will result in an excellent 
program. 


I think the assessment is a good way to say what 
you feel or think. 


Upgrade notification of changes in processes/ 
procedures to ensure everyone needing to know knows 
in a timely manner, rather than "piece mill" 
notices. 


Make the questions in the assessment more 
personalized (level of organization (scope)). 


Good - it gave me some areas that I feel I could 
really improve on. 


Will be a good indicator of program office health. 
Well written, thorough, user friendly, applicable. 


It has good functionality. EFasy to use and 
understand. 


Some of the questions were unclear, relative to the 
answer scale provided. I feel that this assessment 
could have some value if used properly. 


I thought it was a good assessment, with good 
questions. 


Well planned - thought out. 


The assessment needs to ask questions at the 
division/section level and not always refer to the 
project office as a whole. 


Is on the mark. Need good resume function to 


ensure people have plenty of time to answer word 
questions. 


The assessment addressed all disciplines. 


Delete rewards group. The rewards policy is rigid 
and cannot be changed from within. 


Delete technology group. Project is automated. 
Not worth the focus. 


Delete rewards and technology groups. 
Add groups that focus on products and services. 


Change Group 8 from Technology to Products. 
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Task group (#2) Scale: moderately doesn’t apply. 
All of the questions in section 4 should use scale 
2 


Add an area that measures the effect of external 
influences on the project office (PEO and MICOM). 


Add an area or some questions about how well the 
PMO work with the Contractor. 


Define "free" or "open" response in the 
instructions. Suggest "multiple choice" and "short 
answer". (2) 


Most of the questions are to general (encompassing 
the entire PMO). Most people will not have a clue 
at this level. Focus all questions down to the 
personal® or office level. The analysis of all 
responses will give an accurate picture of the 
entire office. 


Barely selection on scale #1 might be better as 
Rarely. 


Consider allowing decimal responses (in between two 
answers). 


Emphasize PMO internal answers. 


Don’t care for open ended questions. Stress 
importance of these questions in instructions. 


I have yet to see the use of the words Value Added 
or Good Enough. Look at incorporating these 
phrases. 


Informal structure incorporates Integrated Product 
Teams (IPT). 


Personalize all questions. 


Open questions need to solicit positive comments as 
well. (2) 


Focus questions to the division, branch level. 


Stress confidentiality of information. Possibly 
incorporate a letter from the PM backing the 
project and stressing confidentiality. 


Asks nothing about our reason for being! Project 
offices supply Products (HW, SW, Tng & Maintenance 
equipment, and documentation to the soldier) and 
Services (technical support, field services, new 
material studies and analysis, service life 
surveillance, depot support, etc.. to soldiers in 
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the field, as well as to HQDA and TRADOC). The 
purpose of this survey is to see how well we are 
doing this job and whether we are on a path of 
continuous improvement. 


e The words Reasonably and Moderately are difficult 
to differentiate between. Consider using alternate 
descriptions. 

® Mix and match selective groups of questions. 

Software: 


Item B5 (part 2): Did you encounter any difficulties using the 
software? If yes, please describe the difficulties. 

° No response . 
Item B6: What, if any, improvements to the software would you 
suggest? 


° Streamline the installation process to eliminate 
the additional "install" and icon setup portions. 
This, however, is not a major problem. Also, you 


might look for a way to disable the comma key 
during text entry so as to eliminate the problems 
with comma delimited files. 


° Add default buttons for the next screen. Add 
infinite scale for answers. 

° Allow the user to back up to previous question. 

° The wait between question groups needs to be 
eliminated. 


Item B7: Please, provide any general comments you may have 
about the software. 


° Software worked well - the windows were 
aesthetically pleasing. The arrow method for 


selecting responses was different, but easy and 
pleasant to use. 


° The software was clear and concise. Made the 
assessment easier to answer in short period of time. 

e I think that the software is great and I really 
enjoyed using it. 

° Some of the bar scales should be the strongly agree 
scale. 
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Instead of dragging the arrow to answer the 
questions, let me click on the answer and go. 


Easy to use. Good choice of color scheme - easy to 
read. 


Has a professional "look". Makes respondents have 
more confidence in results. 


User friendly. Well written. (4) 


Using the windows software is certainly a nice 
innovation. However, the facilitator is obligated 
to provide the general instructions repetitively, 
therefore, more time consuming when compared to all 
respondents at a single session. (Latter half of 
comment retracted when informed about networking 
the software and automating the instructions.) 


Good package. 


Add Senior Programmer to the position listing. 


Include “N/A” or “Other” in all lists. 


Unclear on “time in position” field. Does it refer 
to only to duty in current office or type of duty 
position? 


Need to be able to browse through questions both 
forward and backwards. (17) 


Must be careful when dragging arrow. If you 
release late, it} could change your answer. 
Mention in instructions. 


Need to specify “Years” for time demographic 
fields. (5) 


Might be better to ask for DOB vice age. 


Scale might be easier to use (dragging arrow) if it 
was Sideways on the screen. 


Management vs others is an important comparison for 
analysis. 


Difficult to switch between instructions and 
assessment. 


Need to find a way to have the program disregard 
the commas in the text fields so the user can 
type normally, else find a different way to export 
or analyze the data. (12) 


The year field must be defined. Is it the year you 
are taking the assessment? (2) 
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Need to let the user know that it is the office 
symbol you are looking for in the office field. 


Combine GS and GM to GS/GM in the rank field. 


Is it possible to let the user select an answer 
that is in between those offered on the scale? 


Adjust the main screen to include AGMS and the PEO. 


Add “Logistics Management Specialist” to the 
position list. (2) 


Is the time in the organization field based on 
cumulative time in the organization (what about 
people that have been assigned there twice or 
more?) (2) 


Consider tailoring certain groups of questions to 
be viewed only by certain groups of people. For 


example Management Questions (do not appear to all 
others). 


Add the position Support Clerk to the position 
Lass. 


Does time in organization refer to Government or 
the PMO? (2) 


Add contract specialist to the position list. 
Add below high school in the education field. 
Add Branch Chief to the position list. 


Add years of college, other than degree to the 
education list. (3) 


Configure the program to enable clicking on the 
answers instead of dragging the arrow across the 


screen. (14) 
Add program analyst to the position list. 


Add SFAE-MSL-HD-P to the office listing (scrub the 
office list against the new wire diagram). 


Incorporate an alert message to remind users of 
their respondent number upon exiting the program 
from any place, except the end of the 
questionnaire. 


Add Electronics Engineer to the position list. (2) 
Add Systems Administrator to the position list. 
Consider two configurations: A Stand alone and 

a network version. 
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Add General Engineer to the position list. 


Add Configuration Management Specialist to the 
position listing. (2) 


Consider adding ages greater than 65 to the age 
listing. 


Add Operations Research Analyst to the position 
listing. 


Are you asking for the percent of the office or the 
percent of the individual definitization? 


Add a Core/Matrix demographic field. (5) 
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APPENDIX I: PROJECT MANAGER INTERVIEW 


Name: Organization: 








1. What do you intend to do with the results of the 
assessment? 





























2. What types of analyses of the assessment data would 
be most beneficial to you? 

















3. What types of outputs (presentations, reports, 
etc.) would be most beneficial to you? 








4. How do you define program health? 











5. In your opinion, is the assessment, as written, 
focused on the indicators that determine the program health 
of your organization? 























6. What areas of program health are not covered in the 
assessment? 
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7. At what level(s) do you intend to conduct this 
assessment at in the future? 























8. Do you intend to benchmark your organization 
against other organizations in the future using this 
assessment? If yes, which organizations? 








9. Does the Self-assessment package (assessment and 
software) provide you with the tools necessary to identify 
weaknesses in your organization and monitor their 
improvement ? 
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APPENDIX J: QUALITATIVE RESPONSE SUMMARY: PROJECT MANAGER 
; INTERVIEWS 


Item 1. What do you intend to do with the results of the 


unit self-assessment? 


Establish baseline for comparison of future repetitive 
test responses for the purpose of recognizing trends and 
focusing on quality improvement. 


Evaluate the PMO. Establish trends in the PMO. 


Analyze the results to determine the health of the 
organization. Pinpoint areas to work on, such as 
communications, procedures, etc... 


Item 2. What types of analyses of the assessment data would be 
most beneficial to you? 


Each category (mission, task, etc...) with actual 
results, ability to compare to historical data, trend 
analysis. Discuss accuracy and model sensitivities. 


Comparisons of demographically significant subsets of the 
PMO against the total population and against each other 
(i.e. each division/branch office against the overall 
scores, Management against all others). Benchmark 
against previous years (five years total). 


Big three or four problem areas - what are they? And at 
what levels in the organization - senior leadership, mid- 
level managers, worker bees? Is age versus grade/level 
the biggest differentiation in the answers we get? Are 
"don't knows" a significant percentage of the responses? 


What types of outputs (presentations, reports, etc...) 
would be most beneficial to you? 


Statistics and bar charts. Carry up to five data samples 
(Current plus past four). 


Statistics and bar charts summarizing and comparing the 
data. Current year against previous years. Management 
against all others. Offices against offices and the 
total population. Percentage scored against possible. 
Not concerned with age, education level, or military 
versus Civilian. 


Graphs of answers by rank, age, organization, and 


product. Trend charts - most people that thought "a" was 
screwed up also thought "b" was screwed up. 
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Item 4. 


Item 5. 


Item 6. 


Item 7. 


Item 8. 


How do you define program health? 


Success in fielding and supporting material to the 
soldier. 


Job getting done, people like the job, they have the 
tools they need, the future is reasonably clear, rewards 
are substantial for the deserving, people think the front 
office cares, people think they have authority without 
"mother may I", dead wood gone. 

In your opinion, is the assessment, as written, focused 
on the indicators that determine the program health of 
your organization? 

Not yet. Need to measure success of our ultimate 
products (material and services) . 


For the most part. We have outside influences (PEO 
structure, MICOM, contractors) that aren't really 
addressed. 


What areas of program health are not covered in the 
assessment? 


Products and Services. 
Customer satisfaction, Integrated product Teams (IPT). 


Do the people think the program they are working on (i.e. 
ITAS, IBAS, etc...) are on track? 
At what level(s) do you intend to conduct this 
assessment at in the future? 

All levels in the PMO (100%). (2) 


100% (verbal response extracted from initial guidance) . 


Do you intend to benchmark your organization against 
other organizations in the future using this assessment? 
If yes, which organizations? 

All projects in the PEO, ultimately. Already done once, 
using inappropriate tool. 


Not concerned with benchmarking against other projects. 
Focus is on benchmarking against previous years (verbal 
response extracted from initial guidance). 
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Not necessarily - only several other organizations use 
exactly the same survey. 


Does the Self-assessment package (assessment and 


software) provide you with the tools necessary to 


identify weaknesses in your organization and monitor 
their improvement? 


Yes. 


Depends on what the analyses look like. Too soon to 
tell. 
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APPENDIX K: UNIT FACILITATOR INTERVIEW 


Name: Organization: 





1. Compared to the last assessment you facilitated, to 
what extent did the software change your role as 
facilitator? In what way(s) ? 




















2. Do you think you will be able to manage the data 
collection and analysis process once the software is 
installed on the organization's network and each respondent 
completes the assessment from their individual work 
stations? 























3. What difficulties, if any, do you expect to 
encounter? 

















4. Do you think the software procedures for gathering, 
compiling, exporting, and analyzing data are easier or 
harder than doing these tasks manually? In what way? 

















5. Are the instructions for manipulating data using 
the software complete? Understandable? 




















6. Were you able to import the assessment output file 
into your current analysis program(s) to conduct an analysis 
on the assessment data? What difficulties, if any, did you 
encounter? 























7. Which analysis program(s) do you intend to use with 
this data in the future? 























8. What types of analyses do you intend to conduct on 
this data in the future? 











9. What types of outputs (presentations, reports, 
etc..) do you intend to prepare using this data in the 
future? 





10. Are the instructions for manipulating the 
assessment data using the software complete? 
Understandable? 
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APPENDIX L: QUALITATIVE RESPONSE SUMMARY: UNIT FACILITATOR 


Item 1. 


Item 2. 


Item 3. 


Item 4. 


Item 5. 


INTERVIEWS 


Compared to the last assessment you facilitated, to what 
extent do you expect the software change your role as 
facilitator? In what way(s) ? 

Facilitator much less involved in filling out the survey. 


Keeps facilitator bias out. Much better. More tailored 
to the PMO. Easier on facilitator. 


It will make the analysis of the data easier, especially 
if it is fully automated. It will, however, be difficult 
to manage who has or has not responded to the survey at 
any given time. 
Do you think you will be able to manage the data 
collection and analysis process once the software 
is installed on the organization's network and each 
respondent completes the assessment from their individual 
work stations? 


Should not be too difficult. 
No, it will be extremely difficult to manage who has or 
has not completed the survey based solely on the number 
of records in the database. 

What difficulties, if any, do you expect to encounter? 


Tailoring analyses to detect demographic differences. 


Do you think the software procedures for gathering, 
compiling, exporting, and analyzing data are easier or 
harder than doing these tasks manually? In what way? 


Should be easier than manual. Have not used or viewed 
the analysis sub-program. 


It will be much easier logistically. Also, the compiling 
and analysis of the data should be much easier on the 
facilitator. 


Are the instructions for manipulating data using the 
software complete? Understandable? 
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Item 6. 


Item 7. 


Item 8. 


Item 9. 


Item 10. 


The instructions were not complete at the time of 
testing. 
Are you familiar with the steps involved in importing a 
comma or tab delimited text output file into your current 
analysis program(s) to conduct an analysis on the 


assessment data? What difficulties, if any, did you 
encounter? 


No. (2) 


Which analysis program(s) do you intend to use with this 
data in the future? 


Microsoft Office 


Would wish the analysis be a plotting sub-routine. Don't 
want just raw data. 


MS Office is installed on our computers. Program written 
by the researcher would be fine. 


What types of analyses do you intend to conduct on this 
data in the future? 


Overall (general statistics). Historical comparison by 
item number to determine trends. Comparisons by 
demographic groups, if necessary, to identify, focus in, 


on problem areas. 
That is a decision for top management. 


What types of outputs (presentations, reports, etc..) do 
you intend to prepare using this data in the future? 
Would use plots to brief top management. 
Statistical tables and bar charts. Anything that will 
effectively demonstrate the areas that top management has 
said they want to see. 


Are the instructions for manipulating the assessment data 
using the software complete? Understandable? 


The instructions were not complete at the time of 
testing. 
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How much control do you desire to have over the 
facilitator (DBASE, analysis functions) ? 


Would like a fully automated, canned, analysis program. 


Would like both a canned, fully automatic, analysis 
program and the ability to generate additional queries 
from the database and plot them. 





164 


APPENDIX M: INSTRUCTIONS: THE CCAWS/AGMS UNIT SELF-ASSESSMENT 


1. The CCAWS/AGMS Unit Self-assessment is composed of 77 items 
divided into eleven groups. There are two types of items used in 
the assessment: multiple choice and short answer. Multiple choice 
items are answered by placing the sliding arrow opposite the 
desired response from the selections provided. Each multiple 
choice item is rated on one of the following scales. 





[—rorenty id e+ | strong agree | 


Strongly Disagree 


Don't Know 


Moderately Slightly Disagree 


Note: "Don't Know" is not a neutral response. 


2. The second type of items used are short answer items. The 
value of the assessment is largely dependent upon the quality of 
the responses to these items. These items are answered by typing 
your response into the yellow box provided. 


3. Before you begin the assessment you will be asked to provide 
demographic information about yourself. This information is 
solicited solely to provide information for use in the analysis of 
the assessment items. This information is requested on a volunteer 
basis. 


4. Instructions on the use of the assessment software are included 
as a separate document. 
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APPENDIX N: INSTRUCTIONS: THE CCAWS/AGMS UNIT SELF-ASSESSMENT 
SOFTWARE 


1. General. The CCAWS/AGMS Unit Self-assessment software is a 
prototype application independently developed for the purpose of 
conducting the CCAWS/AGMS Unit Self-assessment. 


System Requirements. 
a. IBM or compatible computer system. 
386SX-25 or faster CPU. 
4MB RAM. 
7.5MB hard drive (free space). 
MS DOS 5.0 or higher. 
MS Windows 3.1 or higher. 
MS Mouse or equivalent pointing device. 
640x480 resolution video driver or higher. 


3. Software Installation. The following procedures are used to 


install the unit assessment software onto your computer system: 


a. Ensure Microsoft Windows is running on your computer. 
Note: This program cannot run outside of the MS Windows 
environment . 

b. Insert the disk labeled Disk 1 into your 3.5" floppy disk 
drive. 

c. From the File menu select Run. 

d. Type the drive letter of your 3.5" floppy disk drive 
followed by Setup.exe, (example - A: Setup.exe), then press enter. 

e. You will be asked if you are sure you want to install the 
unit assessment program onto your computer system. Select "Yes" if 
you wish to proceed with the installation. To terminate the 
installation, select No or Cancel. 

£. You will be prompted to enter the drive letter or network 
volume ID to which you wish to install the unit assessment program 
files. Enter this information in the space provided. 





g. When prompted, insert the requested discs into the 3.5" 
floppy disk drive, then press OK. 

The unit self-assessment setup program creates a directory 
entitled assess under the root directory of the drive or network 
volume you indicated and then transfers the unit assessment program 
files to this directory. The setup program will then create a Unit 
Self-assessment program group and a CCAWS/AGMS Unit Self-assessment 
program item (icon) within the Windows Program Manager. Note: if 
installing this program on a Local Area Network (LAN), this 
procedure must be repeated manually at each workstation from which 
you wish to access to the assessment program. The procedure to 
accomplish this is: 

1. Ensure MS Windows is running on the desired workstation. 
From the File menu, choose New. 

Select Program Group. 

Type Unit Self-Assessment on the title line. 
Press OK. 

Repeat Step 2. 

Select Program Item. 


orn nu FF WN 


Press the Browse button. 


9. Select the directory or volume ID in which you stored the 
assessment program files. 


10. Select the assess directory. 


11. Select the file entitled assess.exe. 
12. Press OK. 


4. Starting the Assessment Program. The unit assessment program 
is started by "double-clicking" the CCAWS/AGMS Unit Self-Assessment 
program icon with the left mouse button. 


5. Beginning a New Assessment. 
a. Start the Unit Assessment Program (Step 4). 


b: Press the Begin Assessment button in the main program 
window. 
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c.. Enter the appropriate demographic information in the 
spaces provided. Note: the demographic entries are fixed response 
items. To choose your response, click with your left mouse button 
on the arrow to the right of the space provided to reveal a drop- 
down listing of available responses. Scroll through the available 
responses by clicking on the up or down arrows to the right of the 
listing until you find the appropriate response. Select the 
response by clicking on it with the left mouse button. 

d. Press the Continue button. 


7. Conducting the Assessment. As stated in the CCAWS/AGMS Unit 
Self-assessment instructions (separate document), the assessment 
incorporates two types of assessment items: multiple choice and 
shot answer. The following procedures are used to respond to the 
types of items used in the assessment. 

a. Multiple Choice. Multiple choice items asks a question, 
or provides a statement, to which there are a limited number of 
fixed responses. These items are accompanied by one of two scales 
(see CCAWS/AGMS Unit Self-assessment instructions) with a blue 
arrow to the right of the scale, positioned across from the Don't 
Know response. There are three methods of selecting the 
appropriate response to this type of question. 

(1) Select the appropriate response using the mouse or 
pointing device. 

(2) Drag the arrow up or down the scale using the mouse 
or pointing device until it is across from the desired response. 

(3) Position the arrow across from the desired response 
by moving it up or down the scale using the arrow buttons on your 
keyboard. 
Note: It is not necessary to move the arrow if the Don't Know 
response is appropriate. 

b. Short Answer. Short answer items allow the respondent to 
input a "free", unrestricted response to the stated item. Ea) 


answer this type of item, simply type the information into the 
yellow box provided. 
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c. Once you have answered each question press the Continue 
button located in the lower right-hand corner of each item window 
to go to the next item. 


Tee Exiting the program. The program can be exited in several 
ways. 

a. Press the Exit button in the main program window. Note: 
this window only appears at the beginning and the end of the 
assessment process. 

b. "Double-click" the minus sign located in the upper left- 
hand corner of any item window. 


c. Choose Quit from the Facilitator menu (Facilitator only). 


8. Resuming a Previous Assessment. The following procedures are 
used to resume a previous assessment. 

a. Start the unit assessment program (Step 4). 

b. Press the Resume Assessment button in the main program 
window. 

c. Enter your Respondent Number in the upper box. IMPORTANT! 
You must use the same respondent number you received when taking 
the initial assessment. This number is provided to you when you 
exit the original assessment. If you cannot remember this number, 
return to the main program window and begin a new assessment. 

d. Press Tab. 

e. Enter the number that corresponds to the item group that 
you would like to begin (i.e. item group 4 = 4). 

£. Enter the number that corresponds to the item that you 
would like to begin (i.e. item 3 = 3). 

£. Press the Resume button. 


g. Resume answering the remaining questions using the 
procedures in step 6. 


170 


9. Facilitator Functions. 
1. Exporting the Data to MS Access for Analysis: 

a. Start the unit assessment Program (Step 4). 

b. Press the Facilitator Functions button in the main 
program window. 

c. Enter the security password, then press enter. 

d. Choose Export in the Facilitator menu. 
Note: the unit self-assessment program exports the demographic and 
item data into a tab delimited text file. The file is entitled 
trnsfr.txt and is placed in the directory containing the unit 
assessment program files. The unit self-assessment program is 
designed to automatically launch MS Access, then import and format 
the data using this program. Users of database and analysis 
programs other than MS Access can import data from this file 
manually, according to the program's procedures. 

Die Data Analysis. Calculations required to analyze the 
collected data are automatically performed as the data are imported 
into MS Access. To perform additional analyses on the data, refer 
to the MS Access users guide. 

3. Presenting the Analyzed Data. The presentation function 
within the data storage, analysis, and presentation application is 
designed to automatically present the graph depicting the overall 
assessment results, by organization and year. The application, as 
delivered, is capable of presenting five separate levels of 
analyzed data. Each level contains greater detail than the 
previous level. To view each successive level of the analysis, 
simply press the push button provided that corresponds to the area 
in which you would like to focus. For example: The initial 
presentation screen provides an overall assessment, depicting the 
results of each unit within the database for each year the 
assessment has been taken. If you would like to view the CCAWS 
organization in more detail, you would depress the CCAWS button. 
If, however, you would like to compare each organization based on 
the 1995 results, you would press the 1995 button. This procedure 
is repeated at each succeeding level of analysis. Analyzed data 


Neg gals 


can be viewed with respect to each organization, office, 


demographic category, and year, or any combination of these areas. 
Copies of each presentation screen can be made by pressing the 


print button, located on each screen. 


1.7.2 


APPENDIX O: DATA SOURCE MATRIX 


Instruction 


Interviews 


Survey 





#3 


#2 


By 
ia 
3 
a 
ay 
ia 
3 
a 


Literature 
Requirement 


Effectiveness 


Question 





Table 16: Data Source Matrix. 
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Measure : 
Research sank Data Literature | Survey | Survey | Survey F Software 
; of : . Interviews : 
Question ‘ Requirement | Review #2 #3 #4 Instruction 
Effectiveness 


Ss 
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